[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:35 dist-test-slave-jt0t krb5kdc[22170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:35 dist-test-slave-jt0t krb5kdc[22170](info): set up 2 sockets
Jun 19 02:49:35 dist-test-slave-jt0t krb5kdc[22170](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 02:49:36.873212 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:49:36 dist-test-slave-jt0t krb5kdc[22170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:49:36.983405 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:41077
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:41077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:37.062047 22183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:37.062321 22183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:37.062356 22183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:37.063542 22183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:37.063583 22183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:37.063602 22183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:37.063619 22183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:37.063633 22183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:37.065161 22183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:41077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:41077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:37.065415 22183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:37.065621 22183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:37.070757 22191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.070822 22189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.070888 22183 server_base.cc:1034] running on GCE node
W20260619 02:49:37.070778 22188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.071269 22183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:37.071571 22183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:37.072705 22183 hybrid_clock.cc:648] HybridClock initialized: now 1781837377072685 us; error 32 us; skew 500 ppm
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:37.075071 22183 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:49:37.075521 22183 webserver.cc:460] Webserver started at http://127.21.164.254:40689/ using document root <none> and password file <none>
I20260619 02:49:37.075723 22183 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:37.075765 22183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:37.075892 22183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:37.076980 22183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "84ade9c1cc1c459f8a7dfeace087a1c3"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "e516b3cc763b5e677a0f076b16ccbc97"
server_key_iv: "f7527502dda4cbaa19ab28a0b8ae2f59"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.077303 22183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "84ade9c1cc1c459f8a7dfeace087a1c3"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "e516b3cc763b5e677a0f076b16ccbc97"
server_key_iv: "f7527502dda4cbaa19ab28a0b8ae2f59"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.078517 22183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:37.079202 22198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.079347 22183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:37.079434 22183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "84ade9c1cc1c459f8a7dfeace087a1c3"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "e516b3cc763b5e677a0f076b16ccbc97"
server_key_iv: "f7527502dda4cbaa19ab28a0b8ae2f59"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.079499 22183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:37.112262 22183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:37.114660 22183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:37.114826 22183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:37.119750 22183 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:41077
I20260619 02:49:37.119796 22250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:41077 every 8 connection(s)
I20260619 02:49:37.120214 22183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:37.121296 22251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:37.124289 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3: Bootstrap starting.
I20260619 02:49:37.125029 22251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:37.125326 22251 log.cc:824] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:37.126042 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3: No bootstrap required, opened a new log
I20260619 02:49:37.127583 22251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ade9c1cc1c459f8a7dfeace087a1c3" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41077 } }
I20260619 02:49:37.127738 22251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:37.127772 22251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ade9c1cc1c459f8a7dfeace087a1c3, State: Initialized, Role: FOLLOWER
I20260619 02:49:37.127946 22251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ade9c1cc1c459f8a7dfeace087a1c3" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41077 } }
I20260619 02:49:37.128026 22251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:37.128053 22251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:37.128093 22251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:37.128150 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 22183
I20260619 02:49:37.128249 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:37.128767 22251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ade9c1cc1c459f8a7dfeace087a1c3" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41077 } }
I20260619 02:49:37.128865 22251 leader_election.cc:302] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84ade9c1cc1c459f8a7dfeace087a1c3; no voters:
I20260619 02:49:37.129019 22251 leader_election.cc:288] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:37.129040 22163 external_mini_cluster.cc:1428] Setting key cf3c99e65c11744d50252d413ce696bd
I20260619 02:49:37.129058 22254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:37.129180 22254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [term 1 LEADER]: Becoming Leader. State: Replica: 84ade9c1cc1c459f8a7dfeace087a1c3, State: Running, Role: LEADER
I20260619 02:49:37.129273 22251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:37.129256 22254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ade9c1cc1c459f8a7dfeace087a1c3" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41077 } }
I20260619 02:49:37.129567 22254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84ade9c1cc1c459f8a7dfeace087a1c3. Latest consensus state: current_term: 1 leader_uuid: "84ade9c1cc1c459f8a7dfeace087a1c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ade9c1cc1c459f8a7dfeace087a1c3" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41077 } } }
I20260619 02:49:37.129626 22254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:37.129824 22255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84ade9c1cc1c459f8a7dfeace087a1c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ade9c1cc1c459f8a7dfeace087a1c3" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41077 } } }
I20260619 02:49:37.129917 22255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3 [sys.catalog]: This master's current role is: LEADER
W20260619 02:49:37.131467 22272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:37.131517 22272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:37.133440 22258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:37.133638 22258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:37.136930 22258 catalog_manager.cc:1347] Generated new cluster ID: 9fc7f13d29d6474f892b51f24f4a18cd
I20260619 02:49:37.136982 22258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:37.149808 22258 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:37.151652 22258 catalog_manager.cc:1504] Loading token signing keys...
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:37.158301 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.129895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.21.164.254:41077)
0619 02:49:37.130045 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.130054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.131170 (+ 1116us) server_negotiation.cc:407] Connection header received
0619 02:49:37.134439 (+ 3269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.134448 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.134509 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.134618 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.146687 (+ 12069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.148142 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.149799 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.150443 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.150585 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.154710 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.154715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.154716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:37.154730 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.156296 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.156743 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.156746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.156746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.156853 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.157095 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.157096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.157096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.157342 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.157500 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.157765 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.157910 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
I20260619 02:49:37.171144 22258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84ade9c1cc1c459f8a7dfeace087a1c3: Generated new TSK 0
I20260619 02:49:37.171380 22258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:49:37.200115 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41077
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:37.312978 22277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:37.313196 22277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:37.313220 22277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:37.314330 22277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:37.314365 22277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:37.314391 22277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:49:37.315804 22277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41077
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:37.315999 22277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:37.316186 22277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:49:37.321125 22277 server_base.cc:1034] running on GCE node
W20260619 02:49:37.321094 22282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.321086 22283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.321094 22285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.321566 22277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:37.321810 22277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:37.322945 22277 hybrid_clock.cc:648] HybridClock initialized: now 1781837377322936 us; error 26 us; skew 500 ppm
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:37.325449 22277 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:49:37.325909 22277 webserver.cc:460] Webserver started at http://127.21.164.193:40207/ using document root <none> and password file <none>
I20260619 02:49:37.326104 22277 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:37.326153 22277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:37.326282 22277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:37.327317 22277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "8633a650ae14418e88c62e6d27b7d3c2"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "e022add7cb8278d6ae65b322c79266a7"
server_key_iv: "387db53ec037c14785417ec3c3424f63"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.327653 22277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "8633a650ae14418e88c62e6d27b7d3c2"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "e022add7cb8278d6ae65b322c79266a7"
server_key_iv: "387db53ec037c14785417ec3c3424f63"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.328938 22277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:37.329641 22292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.329787 22277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:37.329869 22277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "8633a650ae14418e88c62e6d27b7d3c2"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "e022add7cb8278d6ae65b322c79266a7"
server_key_iv: "387db53ec037c14785417ec3c3424f63"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.329941 22277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:37.339282 22277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:37.341218 22277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:37.341353 22277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:37.341562 22277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:37.341850 22277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:37.341884 22277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.341912 22277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:37.341935 22277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.347728 22277 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:39639
I20260619 02:49:37.347776 22405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:39639 every 8 connection(s)
I20260619 02:49:37.348073 22277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:37.355464 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 22277
I20260619 02:49:37.355552 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:37.355726 22163 external_mini_cluster.cc:1428] Setting key ca0887fde1a852fc844f9908edb84c8d
I20260619 02:49:37.357003 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.348871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:60293 (local address 127.21.164.254:41077)
0619 02:49:37.348942 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.348944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.349138 (+ 194us) server_negotiation.cc:407] Connection header received
0619 02:49:37.349612 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.349613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.349624 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.349659 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.350578 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.351081 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.352087 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.352296 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.352361 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.354432 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.354434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.354434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:37.354441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.355526 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.356007 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.356008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.356009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.356069 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.356300 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.356300 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.356301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.356442 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.356508 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.356808 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.356911 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:49:37.357087 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.348843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41077 (local address 127.21.164.193:60293)
0619 02:49:37.348997 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.349048 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:37.349526 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:37.349723 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:37.349726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:37.350288 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:37.350499 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:37.350506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.351146 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.351147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:37.352010 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:37.352013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.352368 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.352369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:37.352462 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.352979 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:37.352988 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:37.354335 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:37.355613 (+ 1278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:37.355620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:37.355623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:37.355932 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:37.356126 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:37.356127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:37.356127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:37.356244 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:37.356517 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:37.356523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:37.356670 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 02:49:37.356774 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 02:49:37.356906 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260619 02:49:37.357501 22406 heartbeater.cc:347] Connected to a master server at 127.21.164.254:41077
I20260619 02:49:37.357599 22406 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:37.357774 22406 heartbeater.cc:511] Master 127.21.164.254:41077 requested a full tablet report, sending...
I20260619 02:49:37.358289 22215 ts_manager.cc:194] Registered new tserver with Master: 8633a650ae14418e88c62e6d27b7d3c2 (127.21.164.193:39639)
I20260619 02:49:37.358883 22215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:60293
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:49:37.378983 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41077
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:37.458889 22411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:37.459086 22411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:37.459108 22411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:37.460242 22411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:37.460284 22411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:37.460310 22411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:49:37.461767 22411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41077
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:37.461992 22411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:37.462211 22411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:37.467068 22419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.467159 22411 server_base.cc:1034] running on GCE node
W20260619 02:49:37.467088 22417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.467269 22416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.467478 22411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:37.467738 22411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:37.468878 22411 hybrid_clock.cc:648] HybridClock initialized: now 1781837377468864 us; error 28 us; skew 500 ppm
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:37.471621 22411 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:49:37.472043 22411 webserver.cc:460] Webserver started at http://127.21.164.194:38939/ using document root <none> and password file <none>
I20260619 02:49:37.472240 22411 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:37.472288 22411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:37.472419 22411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:37.473502 22411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "2a5079ce8e09428580bd517429c6ca70"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "40d49bd7bf5364304b89b0b50c2e55b8"
server_key_iv: "8172a511edd883df7dc8129d3e619664"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.473829 22411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "2a5079ce8e09428580bd517429c6ca70"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "40d49bd7bf5364304b89b0b50c2e55b8"
server_key_iv: "8172a511edd883df7dc8129d3e619664"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.475032 22411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:49:37.475769 22426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.475927 22411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:49:37.476007 22411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "2a5079ce8e09428580bd517429c6ca70"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "40d49bd7bf5364304b89b0b50c2e55b8"
server_key_iv: "8172a511edd883df7dc8129d3e619664"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.476073 22411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:37.488340 22411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:37.490231 22411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:37.490377 22411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:37.490592 22411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:37.490885 22411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:37.490917 22411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.490947 22411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:37.490970 22411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.496762 22411 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:44507
I20260619 02:49:37.496819 22539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:44507 every 8 connection(s)
I20260619 02:49:37.497116 22411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:37.503574 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 22411
I20260619 02:49:37.503666 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:37.503839 22163 external_mini_cluster.cc:1428] Setting key 6afeb1fd95794e1a61a39a9f26047f92
I20260619 02:49:37.505731 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.497791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:54977 (local address 127.21.164.254:41077)
0619 02:49:37.497843 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.497845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.498043 (+ 198us) server_negotiation.cc:407] Connection header received
0619 02:49:37.498501 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.498502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.498511 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.498565 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.499468 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.499945 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.500877 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.501054 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.501101 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.502991 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.502993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.502994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:37.503001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.504352 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.504872 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.504873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.504873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.504926 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.505171 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.505172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.505173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.505270 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.505314 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.505605 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.505668 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:49:37.505996 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.497768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41077 (local address 127.21.164.194:54977)
0619 02:49:37.497922 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.497965 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:37.498430 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:37.498621 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:37.498625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:37.499172 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:37.499397 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:37.499401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.500002 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.500003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:37.500819 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:37.500821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.501111 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.501111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:37.501204 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.501639 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:37.501647 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:37.502900 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:37.504454 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:37.504462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:37.504464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:37.504806 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:37.504987 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:37.504987 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:37.504988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:37.505120 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:37.505320 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:37.505325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:37.505485 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 02:49:37.505595 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:49:37.505764 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260619 02:49:37.506419 22540 heartbeater.cc:347] Connected to a master server at 127.21.164.254:41077
I20260619 02:49:37.506516 22540 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:37.506695 22540 heartbeater.cc:511] Master 127.21.164.254:41077 requested a full tablet report, sending...
I20260619 02:49:37.507040 22215 ts_manager.cc:194] Registered new tserver with Master: 2a5079ce8e09428580bd517429c6ca70 (127.21.164.194:44507)
I20260619 02:49:37.507400 22215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:54977
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:49:37.528100 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41077
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:37.608832 22545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:37.609041 22545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:37.609063 22545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:37.610237 22545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:37.610281 22545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:37.610309 22545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:49:37.611793 22545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41077
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:37.612036 22545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:37.612254 22545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:37.616947 22553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.616994 22551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:37.619729 22550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:37.619813 22545 server_base.cc:1034] running on GCE node
I20260619 02:49:37.620013 22545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:37.620240 22545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:37.621426 22545 hybrid_clock.cc:648] HybridClock initialized: now 1781837377621417 us; error 39 us; skew 500 ppm
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:37.628193 22545 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:49:37.628767 22545 webserver.cc:460] Webserver started at http://127.21.164.195:37413/ using document root <none> and password file <none>
I20260619 02:49:37.629027 22545 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:37.629096 22545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:37.629232 22545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:37.630424 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "f6a4252050cc4f2d88a3318fae3bfdfb"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "e31417952a336bf743a3b6da00d53c08"
server_key_iv: "2ce11b9f88fe97daefce18c89cb30aa4"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.630801 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "f6a4252050cc4f2d88a3318fae3bfdfb"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "e31417952a336bf743a3b6da00d53c08"
server_key_iv: "2ce11b9f88fe97daefce18c89cb30aa4"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.632130 22545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:49:37.632975 22560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.633117 22545 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:49:37.633203 22545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "f6a4252050cc4f2d88a3318fae3bfdfb"
format_stamp: "Formatted at 2026-06-19 02:49:37 on dist-test-slave-jt0t"
server_key: "e31417952a336bf743a3b6da00d53c08"
server_key_iv: "2ce11b9f88fe97daefce18c89cb30aa4"
server_key_version: "encryptionkey@0"
I20260619 02:49:37.633256 22545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:37.658708 22545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:37.660521 22545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:37.660636 22545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:37.660859 22545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:37.661162 22545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:37.661197 22545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.661221 22545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:37.661243 22545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:37.667191 22545 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:39759
I20260619 02:49:37.667248 22673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:39759 every 8 connection(s)
I20260619 02:49:37.667574 22545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:37.673082 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 22545
I20260619 02:49:37.673156 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:37.673291 22163 external_mini_cluster.cc:1428] Setting key c93e3dbf001941dd69899cf02aff1622
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:37.676237 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.668234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:56503 (local address 127.21.164.254:41077)
0619 02:49:37.668310 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.668313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.668479 (+ 166us) server_negotiation.cc:407] Connection header received
0619 02:49:37.668974 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.668975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.668985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.669034 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.669925 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.670403 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.671433 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.671648 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.671723 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.673829 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.673829 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.673830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:37.673844 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.675088 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.675452 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.675453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.675453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.675507 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.675691 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.675691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.675692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.675789 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.675833 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.676088 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.676168 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:49:37.676384 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.668208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41077 (local address 127.21.164.195:56503)
0619 02:49:37.668352 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:37.668388 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:37.668891 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:37.669091 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:37.669094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:37.669656 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:37.669857 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:37.669862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.670492 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.670493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:37.671360 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:37.671363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.671724 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.671724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:37.671824 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.672279 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:37.672289 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:37.673751 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:37.675153 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:37.675158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:37.675160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:37.675381 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:37.675558 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:37.675558 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:37.675559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:37.675650 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:37.675829 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:37.675831 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:37.675951 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 02:49:37.676031 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:49:37.676169 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260619 02:49:37.676857 22674 heartbeater.cc:347] Connected to a master server at 127.21.164.254:41077
I20260619 02:49:37.676946 22674 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:37.677115 22674 heartbeater.cc:511] Master 127.21.164.254:41077 requested a full tablet report, sending...
I20260619 02:49:37.677474 22215 ts_manager.cc:194] Registered new tserver with Master: f6a4252050cc4f2d88a3318fae3bfdfb (127.21.164.195:39759)
I20260619 02:49:37.677803 22215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:56503
I20260619 02:49:37.685478 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:49:37.707878 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.700511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.21.164.193:39639)
0619 02:49:37.700706 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.700712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.700726 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:49:37.700753 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.700755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.700769 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.700841 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.701363 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.702086 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.702762 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.703075 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.703125 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.705338 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.705349 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.705350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:37.705357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.706799 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.707092 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.707094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.707095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.707148 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.707363 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.707364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.707364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.707490 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.707592 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.707720 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.707782 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
W20260619 02:49:37.708273 22320 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33368
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:49:37.729079 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:37.722554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.21.164.193:39639)
0619 02:49:37.722608 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:37.722610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:37.722655 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:49:37.722731 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:37.722732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:37.722745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:37.722790 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:37.723340 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.723837 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.724555 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:37.724824 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:37.724883 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:37.726984 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:37.726986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:37.726986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:37.726995 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:37.728088 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.728381 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.728382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.728383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.728433 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:37.728589 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:37.728590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:37.728590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:37.728722 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:37.728841 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:37.728910 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 02:49:37.728988 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:49:37.729591 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 22277
I20260619 02:49:37.733765 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 22411
I20260619 02:49:37.737562 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 22545
I20260619 02:49:37.741217 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 22183
2026-06-19T02:49:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1818 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22688](info): set up 2 sockets
Jun 19 02:49:37 dist-test-slave-jt0t krb5kdc[22688](info): commencing operation
krb5kdc: starting...
W20260619 02:49:38.691987 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:49:38 dist-test-slave-jt0t krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:49:38.801374 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:37555
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:37555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:38.879426 22701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:38.879642 22701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:38.879673 22701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:38.880861 22701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:38.880904 22701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:38.880923 22701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:38.880937 22701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:38.880954 22701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:38.882767 22701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:37555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:37555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:38.883023 22701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:38.883237 22701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:38.887820 22706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:38.887845 22707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:38.887909 22701 server_base.cc:1034] running on GCE node
W20260619 02:49:38.887823 22709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:38.888253 22701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:38.888531 22701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:38.889690 22701 hybrid_clock.cc:648] HybridClock initialized: now 1781837378889671 us; error 30 us; skew 500 ppm
Jun 19 02:49:38 dist-test-slave-jt0t krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837378, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:38.892060 22701 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:49:38.892449 22701 webserver.cc:460] Webserver started at http://127.21.164.254:42031/ using document root <none> and password file <none>
I20260619 02:49:38.892637 22701 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:38.892714 22701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:38.892810 22701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:38.893800 22701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "05a2c757f2034b1b9c14419a4361205b"
format_stamp: "Formatted at 2026-06-19 02:49:38 on dist-test-slave-jt0t"
server_key: "e6ce4a4c654ca69c57faa365e8452f14"
server_key_iv: "fd68000733473b54a67c1ab1fbce7e27"
server_key_version: "encryptionkey@0"
I20260619 02:49:38.894086 22701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "05a2c757f2034b1b9c14419a4361205b"
format_stamp: "Formatted at 2026-06-19 02:49:38 on dist-test-slave-jt0t"
server_key: "e6ce4a4c654ca69c57faa365e8452f14"
server_key_iv: "fd68000733473b54a67c1ab1fbce7e27"
server_key_version: "encryptionkey@0"
I20260619 02:49:38.895277 22701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:38.895998 22716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:38.896183 22701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:38.896292 22701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "05a2c757f2034b1b9c14419a4361205b"
format_stamp: "Formatted at 2026-06-19 02:49:38 on dist-test-slave-jt0t"
server_key: "e6ce4a4c654ca69c57faa365e8452f14"
server_key_iv: "fd68000733473b54a67c1ab1fbce7e27"
server_key_version: "encryptionkey@0"
I20260619 02:49:38.896350 22701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:38.910310 22701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:38.912261 22701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:38.912369 22701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:38.915935 22701 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:37555
I20260619 02:49:38.915969 22768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:37555 every 8 connection(s)
I20260619 02:49:38.916296 22701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:38.917138 22769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:38.919303 22769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b: Bootstrap starting.
I20260619 02:49:38.919831 22769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:38.920053 22769 log.cc:824] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:38.920746 22769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b: No bootstrap required, opened a new log
I20260619 02:49:38.922245 22769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a2c757f2034b1b9c14419a4361205b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37555 } }
I20260619 02:49:38.922392 22769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:38.922415 22769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a2c757f2034b1b9c14419a4361205b, State: Initialized, Role: FOLLOWER
I20260619 02:49:38.922518 22769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a2c757f2034b1b9c14419a4361205b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37555 } }
I20260619 02:49:38.922570 22769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:38.922597 22769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:38.922637 22769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:38.923146 22769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a2c757f2034b1b9c14419a4361205b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37555 } }
I20260619 02:49:38.923239 22769 leader_election.cc:302] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05a2c757f2034b1b9c14419a4361205b; no voters:
I20260619 02:49:38.923421 22769 leader_election.cc:288] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:38.923470 22772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:38.923646 22772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [term 1 LEADER]: Becoming Leader. State: Replica: 05a2c757f2034b1b9c14419a4361205b, State: Running, Role: LEADER
I20260619 02:49:38.923667 22769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:38.923811 22772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a2c757f2034b1b9c14419a4361205b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37555 } }
I20260619 02:49:38.924193 22774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05a2c757f2034b1b9c14419a4361205b. Latest consensus state: current_term: 1 leader_uuid: "05a2c757f2034b1b9c14419a4361205b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a2c757f2034b1b9c14419a4361205b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37555 } } }
I20260619 02:49:38.924312 22773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05a2c757f2034b1b9c14419a4361205b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a2c757f2034b1b9c14419a4361205b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37555 } } }
I20260619 02:49:38.924342 22774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:38.924400 22773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:38.924808 22781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:38.925236 22781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:38.925668 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 22701
I20260619 02:49:38.925736 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:38.925885 22163 external_mini_cluster.cc:1428] Setting key cce460664f668cb67dd0894fc26f053e
I20260619 02:49:38.926949 22781 catalog_manager.cc:1347] Generated new cluster ID: faae523a48f9488ebf1ad42618d96d71
I20260619 02:49:38.926995 22781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:38.931645 22781 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:38.932127 22781 catalog_manager.cc:1504] Loading token signing keys...
Jun 19 02:49:38 dist-test-slave-jt0t krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:38.937338 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:38.926491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.21.164.254:37555)
0619 02:49:38.926629 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:38.926639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:38.926672 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:49:38.926791 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:38.926799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:38.926856 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:38.927000 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:38.927662 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.929139 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.930202 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:38.930775 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:38.930885 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:38.933640 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:38.933645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:38.933645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:38.933657 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:38.935527 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:38.935892 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:38.935895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:38.935896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:38.935994 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:38.936148 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:38.936149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:38.936149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:38.936425 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:38.936536 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:38.936956 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 02:49:38.937096 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260619 02:49:38.939574 22781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05a2c757f2034b1b9c14419a4361205b: Generated new TSK 0
I20260619 02:49:38.939752 22781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:49:38.971611 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37555
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:39.065685 22795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:39.065887 22795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:39.065909 22795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:39.067062 22795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:39.067106 22795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:39.067132 22795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:49:39.068643 22795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37555
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:39.068915 22795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:39.069128 22795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:39.073855 22803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:39.073872 22801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:39.073855 22800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:39.074205 22795 server_base.cc:1034] running on GCE node
I20260619 02:49:39.074443 22795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:39.074674 22795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:39.075820 22795 hybrid_clock.cc:648] HybridClock initialized: now 1781837379075801 us; error 31 us; skew 500 ppm
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:39.078251 22795 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:49:39.078675 22795 webserver.cc:460] Webserver started at http://127.21.164.193:35329/ using document root <none> and password file <none>
I20260619 02:49:39.078887 22795 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:39.078935 22795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:39.079052 22795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:39.080070 22795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "5fcf9f2ace3948689cc6edea212efe1e"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-jt0t"
server_key: "c348e08a44badad5b3b77d7ff81bf631"
server_key_iv: "a25df6bcd6a21cec6eb0d66a41740749"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.080395 22795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "5fcf9f2ace3948689cc6edea212efe1e"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-jt0t"
server_key: "c348e08a44badad5b3b77d7ff81bf631"
server_key_iv: "a25df6bcd6a21cec6eb0d66a41740749"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.081779 22795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:49:39.082549 22810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:39.082712 22795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:39.082791 22795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "5fcf9f2ace3948689cc6edea212efe1e"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-jt0t"
server_key: "c348e08a44badad5b3b77d7ff81bf631"
server_key_iv: "a25df6bcd6a21cec6eb0d66a41740749"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.082859 22795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:39.098670 22795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:39.100600 22795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:39.100759 22795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:39.100970 22795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:39.101253 22795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:39.101284 22795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:39.101313 22795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:39.101336 22795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:39.106501 22795 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:40961
I20260619 02:49:39.106556 22923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:40961 every 8 connection(s)
I20260619 02:49:39.106846 22795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:39.115806 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.107580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:35293 (local address 127.21.164.254:37555)
0619 02:49:39.107680 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.107682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.107857 (+ 175us) server_negotiation.cc:407] Connection header received
0619 02:49:39.108819 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.108821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.108832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.108879 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:39.109518 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.110018 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.111020 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.111236 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.111293 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.113395 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:39.113396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:39.113397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:39.113405 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:39.114543 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.114898 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.114899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.114899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.114966 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.115175 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.115176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.115176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.115292 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:39.115347 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.115607 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.115703 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:49:39.115942 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.107516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:37555 (local address 127.21.164.193:35293)
0619 02:49:39.107718 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:39.107766 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:39.108707 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:39.108964 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:39.108969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:39.109229 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:39.109423 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.109427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.110100 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.110101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.110932 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.110934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.111305 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.111306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.111433 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.111922 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:39.111930 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:39.113304 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:39.114613 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:39.114619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:39.114622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:39.114832 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:39.115018 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:39.115019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:39.115020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:39.115121 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:39.115342 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:39.115349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:39.115490 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 02:49:39.115577 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 02:49:39.115729 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260619 02:49:39.116400 22924 heartbeater.cc:347] Connected to a master server at 127.21.164.254:37555
I20260619 02:49:39.116487 22924 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:39.116689 22924 heartbeater.cc:511] Master 127.21.164.254:37555 requested a full tablet report, sending...
I20260619 02:49:39.116813 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 22795
I20260619 02:49:39.116925 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:39.117094 22163 external_mini_cluster.cc:1428] Setting key e962caa06e90f0ff999d5755d231dc1b
I20260619 02:49:39.117218 22733 ts_manager.cc:194] Registered new tserver with Master: 5fcf9f2ace3948689cc6edea212efe1e (127.21.164.193:40961)
I20260619 02:49:39.118017 22733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:35293
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:49:39.139396 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37555
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:39.219383 22929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:39.219601 22929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:39.219625 22929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:39.220853 22929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:39.220898 22929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:39.220927 22929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:49:39.222451 22929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37555
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:39.222681 22929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:39.222904 22929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:39.227686 22935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:39.227691 22934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:39.227859 22929 server_base.cc:1034] running on GCE node
W20260619 02:49:39.227725 22937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:39.228166 22929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:39.228406 22929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:39.229542 22929 hybrid_clock.cc:648] HybridClock initialized: now 1781837379229527 us; error 27 us; skew 500 ppm
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:39.231997 22929 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:49:39.232457 22929 webserver.cc:460] Webserver started at http://127.21.164.194:45313/ using document root <none> and password file <none>
I20260619 02:49:39.232705 22929 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:39.232757 22929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:39.232877 22929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:39.233916 22929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "7f8d256785904c6cbeb67d289a498b31"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-jt0t"
server_key: "e0d8c7dd386863beb4bdc88e6ea248fe"
server_key_iv: "26392a41f75c47bb2c3552e821fdac39"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.234264 22929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "7f8d256785904c6cbeb67d289a498b31"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-jt0t"
server_key: "e0d8c7dd386863beb4bdc88e6ea248fe"
server_key_iv: "26392a41f75c47bb2c3552e821fdac39"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.235556 22929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:49:39.236352 22944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:39.236501 22929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:49:39.236583 22929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "7f8d256785904c6cbeb67d289a498b31"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-jt0t"
server_key: "e0d8c7dd386863beb4bdc88e6ea248fe"
server_key_iv: "26392a41f75c47bb2c3552e821fdac39"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.236670 22929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:39.273036 22929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:39.275104 22929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:39.275223 22929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:39.275462 22929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:39.275779 22929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:39.275811 22929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:39.275839 22929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:39.275856 22929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:39.281324 22929 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:39499
I20260619 02:49:39.281399 23057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:39499 every 8 connection(s)
I20260619 02:49:39.281697 22929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:39.284375 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 22929
I20260619 02:49:39.284446 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:39.284590 22163 external_mini_cluster.cc:1428] Setting key caf2edf7124249949e97e2a4448862d4
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:39.291534 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.282389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:46255 (local address 127.21.164.254:37555)
0619 02:49:39.282451 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.282453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.282631 (+ 178us) server_negotiation.cc:407] Connection header received
0619 02:49:39.283496 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.283497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.283506 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.283562 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:39.284153 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.284802 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.286119 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.286311 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.286377 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.288710 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:39.288711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:39.288712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:39.288720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:39.290209 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.290587 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.290588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.290588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.290648 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.290873 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.290874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.290874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.291016 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:39.291085 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.291347 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.291437 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:49:39.291625 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.282358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:37555 (local address 127.21.164.194:46255)
0619 02:49:39.282515 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:39.282555 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:39.283417 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:39.283631 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:39.283634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:39.283899 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:39.284086 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.284091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.284887 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.284888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.286046 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.286050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.286380 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.286381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.286527 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.287072 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:39.287080 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:39.288581 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:39.290293 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:39.290298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:39.290301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:39.290517 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:39.290712 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:39.290713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:39.290713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:39.290816 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:39.291082 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:39.291086 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:39.291222 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 02:49:39.291309 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 02:49:39.291434 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260619 02:49:39.292098 23058 heartbeater.cc:347] Connected to a master server at 127.21.164.254:37555
I20260619 02:49:39.292187 23058 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:39.292368 23058 heartbeater.cc:511] Master 127.21.164.254:37555 requested a full tablet report, sending...
I20260619 02:49:39.292770 22733 ts_manager.cc:194] Registered new tserver with Master: 7f8d256785904c6cbeb67d289a498b31 (127.21.164.194:39499)
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
I20260619 02:49:39.293226 22733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:46255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:49:39.308596 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37555
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:39.386828 23063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:39.387048 23063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:39.387071 23063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:39.388232 23063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:39.388275 23063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:39.388303 23063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:49:39.389849 23063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37555
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:39.390076 23063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:39.390295 23063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:39.395033 23069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:39.395048 23068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:39.395141 23063 server_base.cc:1034] running on GCE node
W20260619 02:49:39.395033 23071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:39.395493 23063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:39.395740 23063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:39.396908 23063 hybrid_clock.cc:648] HybridClock initialized: now 1781837379396890 us; error 32 us; skew 500 ppm
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:39.399364 23063 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:49:39.399818 23063 webserver.cc:460] Webserver started at http://127.21.164.195:34189/ using document root <none> and password file <none>
I20260619 02:49:39.400029 23063 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:39.400072 23063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:39.400164 23063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:39.401201 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "4fc2898874b74680a6dc4465b477e535"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-jt0t"
server_key: "d47e91d66a1492c25fed136940f715f1"
server_key_iv: "f08839fcd82c3cf30fe2162c0e9feaac"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.401487 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "4fc2898874b74680a6dc4465b477e535"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-jt0t"
server_key: "d47e91d66a1492c25fed136940f715f1"
server_key_iv: "f08839fcd82c3cf30fe2162c0e9feaac"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.402637 23063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:39.403254 23078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:39.403404 23063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:39.403479 23063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "4fc2898874b74680a6dc4465b477e535"
format_stamp: "Formatted at 2026-06-19 02:49:39 on dist-test-slave-jt0t"
server_key: "d47e91d66a1492c25fed136940f715f1"
server_key_iv: "f08839fcd82c3cf30fe2162c0e9feaac"
server_key_version: "encryptionkey@0"
I20260619 02:49:39.403543 23063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:39.413549 23063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:39.415216 23063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:39.415347 23063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:39.415547 23063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:39.415836 23063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:39.415871 23063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:39.415899 23063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:39.415921 23063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:39.421236 23063 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:39495
I20260619 02:49:39.421295 23191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:39495 every 8 connection(s)
I20260619 02:49:39.421563 23063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:39.422917 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 23063
I20260619 02:49:39.423028 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:39.423276 22163 external_mini_cluster.cc:1428] Setting key fe54bbfc403eb8e875c739436add3fdb
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:39.430269 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.422207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:49461 (local address 127.21.164.254:37555)
0619 02:49:39.422266 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.422269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.422468 (+ 199us) server_negotiation.cc:407] Connection header received
0619 02:49:39.423553 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.423553 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.423563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.423603 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:39.424344 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.424957 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.425894 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.426076 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.426124 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.428061 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:39.428062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:39.428062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:39.428068 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:39.429266 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.429593 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.429594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.429594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.429637 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.429812 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.429812 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.429812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.429910 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:39.429952 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.430150 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.430205 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:49:39.430449 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.422185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:37555 (local address 127.21.164.195:49461)
0619 02:49:39.422345 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:39.422381 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:39.423464 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:39.423690 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:39.423695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:39.424040 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:39.424261 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.424266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.425020 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.425021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.425833 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.425835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.426130 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.426131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.426225 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.426645 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:39.426653 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:39.427985 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:39.429329 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:39.429334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:39.429336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:39.429538 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:39.429681 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:39.429682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:39.429682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:39.429771 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:39.429949 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:39.429952 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:39.430057 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 02:49:39.430132 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:49:39.430250 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260619 02:49:39.430884 23192 heartbeater.cc:347] Connected to a master server at 127.21.164.254:37555
I20260619 02:49:39.430967 23192 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:39.431130 23192 heartbeater.cc:511] Master 127.21.164.254:37555 requested a full tablet report, sending...
I20260619 02:49:39.431531 22733 ts_manager.cc:194] Registered new tserver with Master: 4fc2898874b74680a6dc4465b477e535 (127.21.164.195:39495)
I20260619 02:49:39.431890 22733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:49461
I20260619 02:49:39.434742 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:39.441390 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.436111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.21.164.254:37555)
0619 02:49:39.436159 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.436161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.436243 (+ 82us) server_negotiation.cc:407] Connection header received
0619 02:49:39.436285 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.436286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.436293 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.436335 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:39.436876 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.437322 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.437956 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.438122 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.438162 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.439154 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:39.439155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:39.439156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:39.439161 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:39.440376 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.440742 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.440743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.440743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.440790 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.440918 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.440919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.440919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.441040 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:39.441110 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.441268 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.441325 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:49:39.443356 22733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:39.444887 22733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:39.451592 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.447130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:39495 (local address 127.0.0.1:55000)
0619 02:49:39.447440 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:39.447479 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:39.447534 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:39.447918 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:39.447918 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:39.447920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:39.447953 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.447954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.449125 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.449125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.450398 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.450399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.451273 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.451273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.451335 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.451345 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:49:39.451457 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:49:39.451510 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":58,"thread_start_us":108,"threads_started":1}
I20260619 02:49:39.451610 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.447208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.21.164.195:39495)
0619 02:49:39.447780 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.447785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.447794 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:39.447825 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.447827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.447841 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.447911 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:39.447992 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.449031 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.450465 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.451203 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.451274 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.451326 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.451491 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.451531 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":65,"threads_started":1}
I20260619 02:49:39.452312 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.447135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:39499 (local address 127.0.0.1:55548)
0619 02:49:39.447492 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:39.447501 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:39.447567 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:39.447735 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:39.447738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:39.447740 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:39.447792 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.447794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.449064 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.449065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.450382 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.450385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.452083 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.452084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.452142 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.452147 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:39.452203 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:49:39.452243 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260619 02:49:39.452319 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.447278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.21.164.194:39499)
0619 02:49:39.447439 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.447444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.447575 (+ 131us) server_negotiation.cc:407] Connection header received
0619 02:49:39.447625 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.447627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.447646 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.447727 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:39.447838 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.448978 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.451363 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.452026 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.452084 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.452134 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.452215 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.452248 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260619 02:49:39.452517 23126 tablet_service.cc:1467] Processing CreateTablet for tablet 8b475a96170c44538efa5e1a49db9cf1 (DEFAULT_TABLE table=test-table [id=cd2fe985e785468da9a42a96fa851ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:39.452798 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.447386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:40961 (local address 127.0.0.1:55912)
0619 02:49:39.447627 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:39.447635 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:39.447670 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:39.449312 (+ 1642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:39.449312 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:39.449315 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:39.449365 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.449367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.450266 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.450267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.451332 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.451335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.452577 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.452578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.452635 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.452639 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:39.452703 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:49:39.452739 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":62,"threads_started":1}
I20260619 02:49:39.452934 23126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b475a96170c44538efa5e1a49db9cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:39.452994 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.447499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.21.164.193:40961)
0619 02:49:39.449175 (+ 1676us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.449179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.449187 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:39.449223 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.449224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.449236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.449296 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:39.449415 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.450179 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.451406 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.452341 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.452564 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.452621 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.452880 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.452921 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1644,"thread_start_us":73,"threads_started":1}
I20260619 02:49:39.453104 22992 tablet_service.cc:1467] Processing CreateTablet for tablet 8b475a96170c44538efa5e1a49db9cf1 (DEFAULT_TABLE table=test-table [id=cd2fe985e785468da9a42a96fa851ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:39.453387 22992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b475a96170c44538efa5e1a49db9cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:39.456236 23209 tablet_bootstrap.cc:492] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535: Bootstrap starting.
I20260619 02:49:39.456481 23210 tablet_bootstrap.cc:492] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31: Bootstrap starting.
I20260619 02:49:39.456557 22858 tablet_service.cc:1467] Processing CreateTablet for tablet 8b475a96170c44538efa5e1a49db9cf1 (DEFAULT_TABLE table=test-table [id=cd2fe985e785468da9a42a96fa851ee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:39.456885 22858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b475a96170c44538efa5e1a49db9cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:39.456938 23209 tablet_bootstrap.cc:654] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:39.457044 23210 tablet_bootstrap.cc:654] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:39.457438 23210 log.cc:824] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:39.457473 23209 log.cc:824] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:39.458343 23209 tablet_bootstrap.cc:492] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535: No bootstrap required, opened a new log
I20260619 02:49:39.458417 23209 ts_tablet_manager.cc:1397] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:39.458489 23210 tablet_bootstrap.cc:492] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31: No bootstrap required, opened a new log
I20260619 02:49:39.458529 23210 ts_tablet_manager.cc:1397] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:39.459816 23210 raft_consensus.cc:348] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8d256785904c6cbeb67d289a498b31" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39499 } } peers { permanent_uuid: "4fc2898874b74680a6dc4465b477e535" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 39495 } } peers { permanent_uuid: "5fcf9f2ace3948689cc6edea212efe1e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40961 } }
I20260619 02:49:39.459981 23210 raft_consensus.cc:374] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:39.460016 23210 raft_consensus.cc:729] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8d256785904c6cbeb67d289a498b31, State: Initialized, Role: FOLLOWER
I20260619 02:49:39.460098 23213 tablet_bootstrap.cc:492] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e: Bootstrap starting.
I20260619 02:49:39.460143 23210 consensus_queue.cc:260] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8d256785904c6cbeb67d289a498b31" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39499 } } peers { permanent_uuid: "4fc2898874b74680a6dc4465b477e535" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 39495 } } peers { permanent_uuid: "5fcf9f2ace3948689cc6edea212efe1e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40961 } }
I20260619 02:49:39.460260 23209 raft_consensus.cc:348] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8d256785904c6cbeb67d289a498b31" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39499 } } peers { permanent_uuid: "4fc2898874b74680a6dc4465b477e535" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 39495 } } peers { permanent_uuid: "5fcf9f2ace3948689cc6edea212efe1e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40961 } }
I20260619 02:49:39.460471 23209 raft_consensus.cc:374] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:39.460517 23209 raft_consensus.cc:729] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc2898874b74680a6dc4465b477e535, State: Initialized, Role: FOLLOWER
I20260619 02:49:39.460565 23210 ts_tablet_manager.cc:1428] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:39.460619 23209 consensus_queue.cc:260] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8d256785904c6cbeb67d289a498b31" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39499 } } peers { permanent_uuid: "4fc2898874b74680a6dc4465b477e535" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 39495 } } peers { permanent_uuid: "5fcf9f2ace3948689cc6edea212efe1e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40961 } }
I20260619 02:49:39.460763 23213 tablet_bootstrap.cc:654] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:39.460759 23058 heartbeater.cc:503] Master 127.21.164.254:37555 was elected leader, sending a full tablet report...
I20260619 02:49:39.460944 23192 heartbeater.cc:503] Master 127.21.164.254:37555 was elected leader, sending a full tablet report...
I20260619 02:49:39.460944 23209 ts_tablet_manager.cc:1428] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 02:49:39.461292 23213 log.cc:824] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:39.462081 23213 tablet_bootstrap.cc:492] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e: No bootstrap required, opened a new log
I20260619 02:49:39.462133 23213 ts_tablet_manager.cc:1397] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:49:39.463361 23213 raft_consensus.cc:348] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8d256785904c6cbeb67d289a498b31" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39499 } } peers { permanent_uuid: "4fc2898874b74680a6dc4465b477e535" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 39495 } } peers { permanent_uuid: "5fcf9f2ace3948689cc6edea212efe1e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40961 } }
I20260619 02:49:39.463485 23213 raft_consensus.cc:374] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:39.463505 23213 raft_consensus.cc:729] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcf9f2ace3948689cc6edea212efe1e, State: Initialized, Role: FOLLOWER
I20260619 02:49:39.463584 23213 consensus_queue.cc:260] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8d256785904c6cbeb67d289a498b31" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39499 } } peers { permanent_uuid: "4fc2898874b74680a6dc4465b477e535" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 39495 } } peers { permanent_uuid: "5fcf9f2ace3948689cc6edea212efe1e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40961 } }
I20260619 02:49:39.463816 23213 ts_tablet_manager.cc:1428] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 02:49:39.463858 22924 heartbeater.cc:503] Master 127.21.164.254:37555 was elected leader, sending a full tablet report...
W20260619 02:49:39.532251 23059 tablet.cc:2369] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:39.607529 22925 tablet.cc:2369] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:39.669298 23214 raft_consensus.cc:482] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:39.669456 23214 raft_consensus.cc:504] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8d256785904c6cbeb67d289a498b31" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39499 } } peers { permanent_uuid: "4fc2898874b74680a6dc4465b477e535" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 39495 } } peers { permanent_uuid: "5fcf9f2ace3948689cc6edea212efe1e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40961 } }
I20260619 02:49:39.669821 23214 leader_election.cc:288] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fc2898874b74680a6dc4465b477e535 (127.21.164.195:39495), 5fcf9f2ace3948689cc6edea212efe1e (127.21.164.193:40961)
W20260619 02:49:39.672134 23193 tablet.cc:2369] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:39.672832 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.669988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:39495 (local address 127.21.164.194:33527)
0619 02:49:39.670297 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:39.670306 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:39.670352 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:39.670461 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:39.670462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:39.670463 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:39.670526 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.670527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.671219 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.671220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.671953 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.671956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.672601 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.672602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.672684 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.672689 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:49:39.672739 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 02:49:39.672770 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":148,"thread_start_us":87,"threads_started":1}
I20260619 02:49:39.672870 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.670065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:33527 (local address 127.21.164.195:39495)
0619 02:49:39.670122 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.670124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.670363 (+ 239us) server_negotiation.cc:407] Connection header received
0619 02:49:39.670394 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.670395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.670405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.670451 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:39.670612 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.671126 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.672051 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.672521 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.672586 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.672595 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.672747 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.672796 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:39.673254 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.669993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:40961 (local address 127.21.164.194:39049)
0619 02:49:39.670041 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:39.670052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:39.670096 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:39.670280 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:39.670281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:39.670282 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:39.670345 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.670347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.671097 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.671098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.671939 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:39.671942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.672516 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.672517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:39.673122 (+ 605us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.673126 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:39.673168 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:49:39.673198 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1}
I20260619 02:49:39.673278 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.670088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:39049 (local address 127.21.164.193:40961)
0619 02:49:39.670136 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.670138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.670146 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:39.670176 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.670177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.670184 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.670242 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:39.670411 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.671011 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.672028 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.672443 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.672498 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.672505 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.673178 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.673218 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:49:39.673285 23146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b475a96170c44538efa5e1a49db9cf1" candidate_uuid: "7f8d256785904c6cbeb67d289a498b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc2898874b74680a6dc4465b477e535" is_pre_election: true
I20260619 02:49:39.673429 23146 raft_consensus.cc:2454] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8d256785904c6cbeb67d289a498b31 in term 0.
I20260619 02:49:39.673573 22878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b475a96170c44538efa5e1a49db9cf1" candidate_uuid: "7f8d256785904c6cbeb67d289a498b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcf9f2ace3948689cc6edea212efe1e" is_pre_election: true
I20260619 02:49:39.673637 22946 leader_election.cc:302] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc2898874b74680a6dc4465b477e535, 7f8d256785904c6cbeb67d289a498b31; no voters:
I20260619 02:49:39.673705 22878 raft_consensus.cc:2454] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8d256785904c6cbeb67d289a498b31 in term 0.
I20260619 02:49:39.673758 23214 raft_consensus.cc:2781] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:39.673797 23214 raft_consensus.cc:482] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:39.673816 23214 raft_consensus.cc:3037] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:39.674530 23214 raft_consensus.cc:504] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8d256785904c6cbeb67d289a498b31" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39499 } } peers { permanent_uuid: "4fc2898874b74680a6dc4465b477e535" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 39495 } } peers { permanent_uuid: "5fcf9f2ace3948689cc6edea212efe1e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40961 } }
I20260619 02:49:39.674654 23214 leader_election.cc:288] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [CANDIDATE]: Term 1 election: Requested vote from peers 4fc2898874b74680a6dc4465b477e535 (127.21.164.195:39495), 5fcf9f2ace3948689cc6edea212efe1e (127.21.164.193:40961)
I20260619 02:49:39.674793 23146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b475a96170c44538efa5e1a49db9cf1" candidate_uuid: "7f8d256785904c6cbeb67d289a498b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc2898874b74680a6dc4465b477e535"
I20260619 02:49:39.674847 22878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b475a96170c44538efa5e1a49db9cf1" candidate_uuid: "7f8d256785904c6cbeb67d289a498b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcf9f2ace3948689cc6edea212efe1e"
I20260619 02:49:39.674866 23146 raft_consensus.cc:3037] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:39.674914 22878 raft_consensus.cc:3037] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:39.675666 22878 raft_consensus.cc:2454] T 8b475a96170c44538efa5e1a49db9cf1 P 5fcf9f2ace3948689cc6edea212efe1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8d256785904c6cbeb67d289a498b31 in term 1.
I20260619 02:49:39.675665 23146 raft_consensus.cc:2454] T 8b475a96170c44538efa5e1a49db9cf1 P 4fc2898874b74680a6dc4465b477e535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8d256785904c6cbeb67d289a498b31 in term 1.
I20260619 02:49:39.675820 22946 leader_election.cc:302] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc2898874b74680a6dc4465b477e535, 7f8d256785904c6cbeb67d289a498b31; no voters:
I20260619 02:49:39.675892 23214 raft_consensus.cc:2781] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:39.676026 23214 raft_consensus.cc:686] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8d256785904c6cbeb67d289a498b31, State: Running, Role: LEADER
I20260619 02:49:39.676100 23214 consensus_queue.cc:237] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8d256785904c6cbeb67d289a498b31" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39499 } } peers { permanent_uuid: "4fc2898874b74680a6dc4465b477e535" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 39495 } } peers { permanent_uuid: "5fcf9f2ace3948689cc6edea212efe1e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40961 } }
I20260619 02:49:39.676908 22733 catalog_manager.cc:5495] T 8b475a96170c44538efa5e1a49db9cf1 P 7f8d256785904c6cbeb67d289a498b31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8d256785904c6cbeb67d289a498b31 (127.21.164.194). New cstate: current_term: 1 leader_uuid: "7f8d256785904c6cbeb67d289a498b31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f8d256785904c6cbeb67d289a498b31" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc2898874b74680a6dc4465b477e535" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 39495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fcf9f2ace3948689cc6edea212efe1e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40961 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:49:39.737564 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.730827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.21.164.193:40961)
0619 02:49:39.730914 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.730916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.730961 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:49:39.731007 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.731008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.731018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.731065 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:39.731560 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.732036 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.732721 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.732900 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.732945 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.735013 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:39.735024 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:39.735024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:39.735031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:39.736587 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.736932 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.736935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.736935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.737004 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.737151 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.737152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.737152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.737282 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:39.737380 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.737425 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.737484 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:49:39.758685 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.752376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.21.164.193:40961)
0619 02:49:39.752442 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.752444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.752473 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:49:39.752540 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.752541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.752550 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.752598 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:39.753130 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.753616 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.754269 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.754449 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.754492 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.756481 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:39.756482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:39.756483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:39.756489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:39.757647 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.757952 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.757953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.757953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.758003 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.758156 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.758157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.758157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.758266 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:39.758349 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.758518 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.758594 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:49:39.778820 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:39.772781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.21.164.193:40961)
0619 02:49:39.772865 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:39.772868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:39.772900 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:49:39.772964 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:39.772964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:39.772974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:39.773021 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:39.773535 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.773985 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.774656 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:39.774828 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:39.774875 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:39.776854 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:39.776856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:39.776856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:39.776867 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:39.777937 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.778223 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.778224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.778224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.778273 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:39.778402 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:39.778402 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:39.778403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:39.778484 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:39.778550 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:39.778695 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 02:49:39.778750 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 02:49:39.779289 22838 tablet_service.cc:2908] Rejecting scan request for tablet 8b475a96170c44538efa5e1a49db9cf1: Uninitialized: clean time has not yet been initialized
I20260619 02:49:39.780233 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 22795
I20260619 02:49:39.784566 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 22929
I20260619 02:49:39.788980 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 23063
I20260619 02:49:39.793326 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 22701
2026-06-19T02:49:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2052 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[23233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[23233](info): set up 2 sockets
Jun 19 02:49:39 dist-test-slave-jt0t krb5kdc[23233](info): commencing operation
krb5kdc: starting...
W20260619 02:49:40.748095 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:49:40 dist-test-slave-jt0t krb5kdc[23233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:49:40.860453 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45781
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:45781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:40.939841 23246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:40.940080 23246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:40.940114 23246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:40.941403 23246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:40.941450 23246 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:49:40.941468 23246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:40.941485 23246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:40.941500 23246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:40.941519 23246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:40.943041 23246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:45781
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:40.943302 23246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:40.943521 23246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:49:40.948349 23246 server_base.cc:1034] running on GCE node
W20260619 02:49:40.948343 23254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:40.948361 23251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:40.948343 23252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:40.948876 23246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:40.949162 23246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:40.950309 23246 hybrid_clock.cc:648] HybridClock initialized: now 1781837380950292 us; error 27 us; skew 500 ppm
Jun 19 02:49:40 dist-test-slave-jt0t krb5kdc[23233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837380, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:40.952585 23246 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:49:40.953012 23246 webserver.cc:460] Webserver started at http://127.21.164.254:38901/ using document root <none> and password file <none>
I20260619 02:49:40.953223 23246 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:40.953270 23246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:40.953394 23246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:40.954419 23246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "d8a689a67f1d46faa5eb1a9617c51f32"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-jt0t"
server_key: "cc47e85ab40610f472baceb3cbc3833d"
server_key_iv: "6d605f88cb2020bead6d4d5c6b25f83c"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.954744 23246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "d8a689a67f1d46faa5eb1a9617c51f32"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-jt0t"
server_key: "cc47e85ab40610f472baceb3cbc3833d"
server_key_iv: "6d605f88cb2020bead6d4d5c6b25f83c"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.955884 23246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 02:49:40.956538 23261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:40.956727 23246 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:40.956809 23246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "d8a689a67f1d46faa5eb1a9617c51f32"
format_stamp: "Formatted at 2026-06-19 02:49:40 on dist-test-slave-jt0t"
server_key: "cc47e85ab40610f472baceb3cbc3833d"
server_key_iv: "6d605f88cb2020bead6d4d5c6b25f83c"
server_key_version: "encryptionkey@0"
I20260619 02:49:40.956874 23246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:40.980548 23246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:40.982336 23246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:40.982472 23246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:40.986830 23246 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45781
I20260619 02:49:40.986876 23323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45781 every 8 connection(s)
I20260619 02:49:40.987187 23246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:40.987833 23324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:40.990144 23324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32: Bootstrap starting.
I20260619 02:49:40.990703 23324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:40.990954 23324 log.cc:824] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:40.991664 23324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32: No bootstrap required, opened a new log
I20260619 02:49:40.993738 23324 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a689a67f1d46faa5eb1a9617c51f32" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45781 } }
I20260619 02:49:40.993875 23324 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:40.993894 23324 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a689a67f1d46faa5eb1a9617c51f32, State: Initialized, Role: FOLLOWER
I20260619 02:49:40.994026 23324 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a689a67f1d46faa5eb1a9617c51f32" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45781 } }
I20260619 02:49:40.994095 23324 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:40.994119 23324 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:40.994164 23324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:40.994678 23324 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a689a67f1d46faa5eb1a9617c51f32" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45781 } }
I20260619 02:49:40.994764 23324 leader_election.cc:302] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8a689a67f1d46faa5eb1a9617c51f32; no voters:
I20260619 02:49:40.994910 23324 leader_election.cc:288] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:40.994949 23327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:40.995113 23327 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [term 1 LEADER]: Becoming Leader. State: Replica: d8a689a67f1d46faa5eb1a9617c51f32, State: Running, Role: LEADER
I20260619 02:49:40.995157 23324 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:40.995184 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 23246
I20260619 02:49:40.995268 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:40.995251 23327 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a689a67f1d46faa5eb1a9617c51f32" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45781 } }
I20260619 02:49:40.995437 22163 external_mini_cluster.cc:1428] Setting key e66dc2709e2c3ade5890e499e1e9a917
I20260619 02:49:40.995564 23329 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8a689a67f1d46faa5eb1a9617c51f32. Latest consensus state: current_term: 1 leader_uuid: "d8a689a67f1d46faa5eb1a9617c51f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a689a67f1d46faa5eb1a9617c51f32" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45781 } } }
I20260619 02:49:40.995565 23328 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8a689a67f1d46faa5eb1a9617c51f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a689a67f1d46faa5eb1a9617c51f32" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45781 } } }
I20260619 02:49:40.995657 23329 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:40.995678 23328 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:40.995862 23334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:40.996395 23334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:40.998023 23334 catalog_manager.cc:1347] Generated new cluster ID: a3dda1d915ac4a66bf3fa5f1f92c6d9e
I20260619 02:49:40.998071 23334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:49:40 dist-test-slave-jt0t krb5kdc[23233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:41.007522 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:40.996033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.21.164.254:45781)
0619 02:49:40.996305 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:40.996315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:40.996833 (+ 518us) server_negotiation.cc:407] Connection header received
0619 02:49:40.997099 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:40.997107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:40.997166 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:40.997311 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:40.998023 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:40.999392 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.000458 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.001079 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.001171 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.003925 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.003930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.003930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.003944 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.005861 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.006256 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.006259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.006260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.006387 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.006555 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.006556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.006556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.006893 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.007062 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.007128 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.007266 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":51,"threads_started":1}
I20260619 02:49:41.008499 23334 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:41.009016 23334 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:41.015256 23334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32: Generated new TSK 0
I20260619 02:49:41.015437 23334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:49:41.045073 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45781
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:41.125456 23350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:41.125661 23350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:41.125684 23350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:41.126823 23350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:41.126858 23350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:41.126878 23350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:41.126896 23350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:49:41.128293 23350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.164.254:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:41.128484 23350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:41.128705 23350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:41.133462 23356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:41.133477 23358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:41.133458 23355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:41.133680 23350 server_base.cc:1034] running on GCE node
I20260619 02:49:41.133908 23350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:41.134130 23350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:41.135275 23350 hybrid_clock.cc:648] HybridClock initialized: now 1781837381135256 us; error 35 us; skew 500 ppm
Jun 19 02:49:41 dist-test-slave-jt0t krb5kdc[23233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.137755 23350 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:49:41.138157 23350 webserver.cc:460] Webserver started at http://127.21.164.193:36595/ using document root <none> and password file <none>
I20260619 02:49:41.138347 23350 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:41.138413 23350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:41.138537 23350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:41.139545 23350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "96750281b69941fe9a00dab6e287bd6a"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-jt0t"
server_key: "3ba7d97dd8db60dedcf360a20a2b1fd7"
server_key_iv: "894078f8060996187b1724790de48962"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.139863 23350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "96750281b69941fe9a00dab6e287bd6a"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-jt0t"
server_key: "3ba7d97dd8db60dedcf360a20a2b1fd7"
server_key_iv: "894078f8060996187b1724790de48962"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.141155 23350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:41.141875 23365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.142036 23350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:41.142117 23350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "96750281b69941fe9a00dab6e287bd6a"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-jt0t"
server_key: "3ba7d97dd8db60dedcf360a20a2b1fd7"
server_key_iv: "894078f8060996187b1724790de48962"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.142175 23350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:41.165202 23350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:41.167002 23350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:41.167140 23350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:41.167652 23350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:41.167696 23350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.167726 23350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:41.167743 23350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.174235 23350 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:37427
I20260619 02:49:41.174398 23480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:37427 every 8 connection(s)
I20260619 02:49:41.174649 23350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:49:41 dist-test-slave-jt0t krb5kdc[23233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:41.177513 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.168152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:53723 (local address 127.21.164.254:45781)
0619 02:49:41.168213 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.168215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.168435 (+ 220us) server_negotiation.cc:407] Connection header received
0619 02:49:41.168995 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.168996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.169013 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.169071 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.169982 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.170749 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.172278 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.172565 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.172620 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.175212 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.175214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.175214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.175223 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.176340 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.176726 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.176727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.176728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.176774 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.176972 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.176973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.176973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.177072 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.177114 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.177354 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.177447 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:49:41.177646 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.168161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45781 (local address 127.21.164.193:53723)
0619 02:49:41.168315 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.168352 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.168898 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.169180 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.169185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.169557 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:41.169878 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.169886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.170855 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.170857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.172182 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.172186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.172668 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.172670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.172824 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.173454 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:41.173464 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:41.175139 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:41.176420 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.176426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.176428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.176639 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.176820 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.176821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.176821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.176917 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.177112 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:41.177115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:41.177261 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.177336 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.177458 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260619 02:49:41.178115 23481 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45781
I20260619 02:49:41.178206 23481 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:41.178390 23481 heartbeater.cc:511] Master 127.21.164.254:45781 requested a full tablet report, sending...
I20260619 02:49:41.178857 23278 ts_manager.cc:194] Registered new tserver with Master: 96750281b69941fe9a00dab6e287bd6a (127.21.164.193:37427)
I20260619 02:49:41.179630 23278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:53723
I20260619 02:49:41.179819 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 23350
I20260619 02:49:41.179916 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:41.180078 22163 external_mini_cluster.cc:1428] Setting key 118df357f2f14af4f6d94a88200135fd
I20260619 02:49:41.184454 23487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.179055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45781 (local address 127.21.164.193:55593)
0619 02:49:41.179160 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.179170 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.179220 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.179332 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.179333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.179477 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:41.179524 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.179526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.180354 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.180355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.180980 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.180982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.181311 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.181312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.181382 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.181815 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:41.181819 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:41.182075 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:41.183515 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.183516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.183516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.183666 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.183855 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.183856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.183856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.183903 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.184167 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:41.184169 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:41.184214 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.184278 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.184379 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":57,"threads_started":1}
I20260619 02:49:41.184482 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.179122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:55593 (local address 127.21.164.254:45781)
0619 02:49:41.179161 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.179163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.179211 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:49:41.179254 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.179255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.179264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.179329 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.179575 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.180250 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.181054 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.181253 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.181308 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.182152 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.182153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.182154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.182161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.183440 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.183733 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.183734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.183734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.183794 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.183955 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.183955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.183956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.184080 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.184143 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.184312 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.184400 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:49:41.204285 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45781
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:41.283640 23490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:41.283851 23490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:41.283875 23490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:41.285068 23490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:41.285112 23490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:41.285131 23490 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:41.285147 23490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:49:41.286566 23490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.164.254:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:41.286796 23490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:41.287014 23490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:41.291893 23496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:41.291966 23490 server_base.cc:1034] running on GCE node
W20260619 02:49:41.291919 23498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:41.291893 23495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:41.292254 23490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:41.292470 23490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:41.293601 23490 hybrid_clock.cc:648] HybridClock initialized: now 1781837381293587 us; error 29 us; skew 500 ppm
Jun 19 02:49:41 dist-test-slave-jt0t krb5kdc[23233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.295966 23490 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:49:41.296419 23490 webserver.cc:460] Webserver started at http://127.21.164.194:34611/ using document root <none> and password file <none>
I20260619 02:49:41.296605 23490 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:41.296666 23490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:41.296775 23490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:41.297789 23490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "fe06a14733cd48babfb611bd858b98cd"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-jt0t"
server_key: "f7be2d8967bad3228c148b01b4ba5ddf"
server_key_iv: "1053ae4e6481154eaaa44dc642ece865"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.298084 23490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "fe06a14733cd48babfb611bd858b98cd"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-jt0t"
server_key: "f7be2d8967bad3228c148b01b4ba5ddf"
server_key_iv: "1053ae4e6481154eaaa44dc642ece865"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.299315 23490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:41.299953 23505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.300086 23490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:41.300158 23490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "fe06a14733cd48babfb611bd858b98cd"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-jt0t"
server_key: "f7be2d8967bad3228c148b01b4ba5ddf"
server_key_iv: "1053ae4e6481154eaaa44dc642ece865"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.300222 23490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:41.309592 23490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:41.311554 23490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:41.311683 23490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:41.312270 23490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:41.312328 23490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.312368 23490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:41.312386 23490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:49:41 dist-test-slave-jt0t krb5kdc[23233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:41.319250 23490 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:37971
I20260619 02:49:41.319298 23620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:37971 every 8 connection(s)
I20260619 02:49:41.319656 23490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:41.321388 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.312617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:55435 (local address 127.21.164.254:45781)
0619 02:49:41.312691 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.312693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.312971 (+ 278us) server_negotiation.cc:407] Connection header received
0619 02:49:41.313607 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.313608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.313618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.313659 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.314527 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.315028 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.316053 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.316239 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.316294 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.318664 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.318665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.318665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.318674 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.320275 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.320621 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.320621 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.320622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.320707 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.320895 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.320896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.320896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.320999 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.321044 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.321240 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.321298 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:49:41.321529 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.312620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45781 (local address 127.21.164.194:55435)
0619 02:49:41.312828 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.312876 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.313508 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.313745 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.313750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.314117 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:41.314412 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.314420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.315109 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.315110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.315969 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.315971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.316306 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.316307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.316431 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.316971 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:41.316982 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:41.318566 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:41.320349 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.320352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.320354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.320562 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.320756 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.320757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.320757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.320853 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.321034 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:41.321036 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:41.321149 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.321224 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.321349 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":118,"threads_started":1}
I20260619 02:49:41.321931 23621 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45781
I20260619 02:49:41.322011 23621 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:41.322178 23621 heartbeater.cc:511] Master 127.21.164.254:45781 requested a full tablet report, sending...
I20260619 02:49:41.322540 23277 ts_manager.cc:194] Registered new tserver with Master: fe06a14733cd48babfb611bd858b98cd (127.21.164.194:37971)
I20260619 02:49:41.323005 23277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:55435
I20260619 02:49:41.327687 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.322936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:33025 (local address 127.21.164.254:45781)
0619 02:49:41.323120 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.323122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.323130 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:41.323146 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.323147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.323155 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.323194 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.323533 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.324002 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.324714 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.324898 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.324953 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.325715 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.325717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.325717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.325724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.326693 (+ 969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.326978 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.326980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.326980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.327031 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.327184 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.327185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.327185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.327330 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.327386 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.327520 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.327591 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260619 02:49:41.327701 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.322883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45781 (local address 127.21.164.194:33025)
0619 02:49:41.322997 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.323007 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.323050 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.323205 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.323205 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.323396 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:41.323462 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.323464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.324084 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.324085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.324593 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.324594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.324956 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.324957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.325030 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.325389 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:41.325393 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:41.325641 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:41.326760 (+ 1119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.326761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.326762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.326921 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.327080 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.327081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.327081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.327137 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.327386 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:41.327387 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:41.327432 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.327493 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.327592 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
I20260619 02:49:41.328994 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 23490
I20260619 02:49:41.329083 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:41.329237 22163 external_mini_cluster.cc:1428] Setting key dd9407a34d90f908a63ea12b9e9077f5
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:49:41.351387 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45781
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:41.430284 23630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:41.430497 23630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:41.430519 23630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:41.431747 23630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:41.431795 23630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:41.431815 23630 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:41.431833 23630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:49:41.433306 23630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.164.254:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:41.433529 23630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:41.433739 23630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:41.438403 23635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:41.438403 23638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:41.438482 23636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:41.438750 23630 server_base.cc:1034] running on GCE node
I20260619 02:49:41.438974 23630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:41.439203 23630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:41.440369 23630 hybrid_clock.cc:648] HybridClock initialized: now 1781837381440352 us; error 28 us; skew 500 ppm
Jun 19 02:49:41 dist-test-slave-jt0t krb5kdc[23233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:41.442798 23630 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:49:41.443215 23630 webserver.cc:460] Webserver started at http://127.21.164.195:41615/ using document root <none> and password file <none>
I20260619 02:49:41.443389 23630 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:41.443428 23630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:41.443518 23630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:41.444514 23630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-jt0t"
server_key: "4f4c720e4c84ecd8973b4ed594c2b0e5"
server_key_iv: "7e5c1a6760a7de4c6881e3baeb347602"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.444893 23630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-jt0t"
server_key: "4f4c720e4c84ecd8973b4ed594c2b0e5"
server_key_iv: "7e5c1a6760a7de4c6881e3baeb347602"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.446143 23630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:41.446780 23645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.446918 23630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:41.446990 23630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd"
format_stamp: "Formatted at 2026-06-19 02:49:41 on dist-test-slave-jt0t"
server_key: "4f4c720e4c84ecd8973b4ed594c2b0e5"
server_key_iv: "7e5c1a6760a7de4c6881e3baeb347602"
server_key_version: "encryptionkey@0"
I20260619 02:49:41.447052 23630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:41.472330 23630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:41.474138 23630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:41.474262 23630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:41.474769 23630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:41.474809 23630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.474838 23630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:41.474852 23630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:49:41 dist-test-slave-jt0t krb5kdc[23233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:41.481359 23630 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:46063
I20260619 02:49:41.481487 23760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:46063 every 8 connection(s)
I20260619 02:49:41.481742 23630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:41.484133 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.475216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:44149 (local address 127.21.164.254:45781)
0619 02:49:41.475283 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.475286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.475465 (+ 179us) server_negotiation.cc:407] Connection header received
0619 02:49:41.475938 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.475939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.475950 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.476019 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.476832 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.477590 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.478676 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.478981 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.479056 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.481603 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.481605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.481605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.481613 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.482908 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.483256 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.483258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.483258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.483316 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.483527 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.483528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.483529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.483645 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.483700 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.483959 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.484044 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:49:41.484287 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.475180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45781 (local address 127.21.164.195:44149)
0619 02:49:41.475321 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.475361 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.475846 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.476098 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.476102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.476457 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:41.476756 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.476762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.477702 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.477703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.478611 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.478613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.479081 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.479082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.479220 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.479717 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:41.479726 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:41.481505 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:41.482979 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.482985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.482987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.483193 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.483366 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.483367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.483367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.483474 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.483696 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:41.483701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:41.483833 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.483926 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.484076 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260619 02:49:41.484736 23761 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45781
I20260619 02:49:41.484815 23761 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:41.484989 23761 heartbeater.cc:511] Master 127.21.164.254:45781 requested a full tablet report, sending...
I20260619 02:49:41.485318 23278 ts_manager.cc:194] Registered new tserver with Master: 5c6cfe7b877e41f4bfb3e56adf8c33cd (127.21.164.195:46063)
I20260619 02:49:41.485852 23278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:44149
I20260619 02:49:41.485991 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 23630
I20260619 02:49:41.486093 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:41.486281 22163 external_mini_cluster.cc:1428] Setting key 6566582466aec6f2bd1164ffbee89acf
I20260619 02:49:41.487574 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:41.491104 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.485704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:43163 (local address 127.21.164.254:45781)
0619 02:49:41.485748 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.485749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.485807 (+ 58us) server_negotiation.cc:407] Connection header received
0619 02:49:41.485874 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.485875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.485884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.485935 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.486318 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.486829 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.487618 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.487846 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.487895 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.488784 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.488785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.488785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.488791 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.490184 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.490486 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.490487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.490487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.490534 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.490711 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.490711 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.490712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.490804 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.490860 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.490988 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.491043 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:49:41.491228 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.485646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45781 (local address 127.21.164.195:43163)
0619 02:49:41.485753 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.485763 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.485821 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.485980 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.485981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.486170 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:41.486231 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.486233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.486991 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.486992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.487548 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.487550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.487908 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.487909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.487985 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.488351 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:41.488355 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:41.488717 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:41.490265 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.490266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.490266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.490434 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.490599 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.490599 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.490600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.490660 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.490854 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:41.490855 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:41.490906 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.490971 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.491068 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":61,"threads_started":1}
Jun 19 02:49:41 dist-test-slave-jt0t krb5kdc[23233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:49:41 dist-test-slave-jt0t krb5kdc[23233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:41.509686 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.503551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.21.164.254:45781)
0619 02:49:41.503607 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.503609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.503636 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:49:41.503660 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.503660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.503681 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.503719 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.504271 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.504862 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.505554 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.505740 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.505798 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.507822 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.507824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.507824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.507831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.508897 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.509169 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.509169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.509169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.509213 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.509334 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.509334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.509334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.509424 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.509493 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.509561 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.509615 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:41.511130 23278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:41.512714 23278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:41.518608 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.514711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:37971 (local address 127.0.0.1:49396)
0619 02:49:41.515006 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.515017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.515070 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.515612 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.515612 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.515614 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:41.515657 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.515658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.516466 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.516467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.517460 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.517463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.518284 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.518285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.518373 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.518389 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.518464 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.518519 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260619 02:49:41.518781 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.514924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:37427 (local address 127.0.0.1:48682)
0619 02:49:41.515075 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.515082 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.515117 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.515416 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.515417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.515419 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:41.515459 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.515460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.516820 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.516821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.517885 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.517887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.518605 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.518606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.518660 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.518664 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.518692 (+ 28us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.518727 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":49,"threads_started":1}
I20260619 02:49:41.518944 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.514711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:46063 (local address 127.0.0.1:42440)
0619 02:49:41.514882 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.514923 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.514982 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.515165 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.515168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.515170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:41.515236 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.515238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.516482 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.516484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.517768 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.517771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.518698 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.518699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.518776 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.518779 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.518835 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.518879 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":62,"thread_start_us":65,"threads_started":1}
I20260619 02:49:41.518628 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.514853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49396 (local address 127.21.164.194:37971)
0619 02:49:41.515296 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.515300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.515308 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:41.515343 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.515345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.515357 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.515554 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:41.515711 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.516391 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.517545 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.518199 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.518276 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.518332 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.518485 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.518530 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":65,"threads_started":1}
I20260619 02:49:41.519127 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.515098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.21.164.193:37427)
0619 02:49:41.515243 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.515248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.515258 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:41.515294 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.515296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.515311 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.515420 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:41.515544 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.516720 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.517962 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.518522 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.518899 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.518965 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.518988 (+ 23us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.519032 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260619 02:49:41.519127 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.514798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.21.164.195:46063)
0619 02:49:41.515014 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.515020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.515034 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:49:41.515072 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.515074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.515090 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.515165 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:41.515288 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.516380 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.517850 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.518547 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.518614 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.518663 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.518939 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.519038 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
I20260619 02:49:41.519753 23695 tablet_service.cc:1467] Processing CreateTablet for tablet 56ba827374614ad485e2f0d6a9e4f0b9 (DEFAULT_TABLE table=test-table [id=c1293c6cf31e4952a16913ab1b85b862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:41.519840 23415 tablet_service.cc:1467] Processing CreateTablet for tablet 56ba827374614ad485e2f0d6a9e4f0b9 (DEFAULT_TABLE table=test-table [id=c1293c6cf31e4952a16913ab1b85b862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:41.519901 23555 tablet_service.cc:1467] Processing CreateTablet for tablet 56ba827374614ad485e2f0d6a9e4f0b9 (DEFAULT_TABLE table=test-table [id=c1293c6cf31e4952a16913ab1b85b862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:41.520148 23415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ba827374614ad485e2f0d6a9e4f0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:41.520149 23695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ba827374614ad485e2f0d6a9e4f0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:41.520231 23555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ba827374614ad485e2f0d6a9e4f0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:41.523224 23781 tablet_bootstrap.cc:492] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: Bootstrap starting.
I20260619 02:49:41.523686 23782 tablet_bootstrap.cc:492] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Bootstrap starting.
I20260619 02:49:41.523948 23781 tablet_bootstrap.cc:654] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:41.524279 23781 log.cc:824] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:41.524623 23782 tablet_bootstrap.cc:654] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:41.524963 23782 log.cc:824] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:41.525040 23781 tablet_bootstrap.cc:492] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: No bootstrap required, opened a new log
I20260619 02:49:41.525103 23781 ts_tablet_manager.cc:1397] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:41.525667 23782 tablet_bootstrap.cc:492] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: No bootstrap required, opened a new log
I20260619 02:49:41.525734 23782 ts_tablet_manager.cc:1397] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 02:49:41.527083 23781 raft_consensus.cc:348] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } }
I20260619 02:49:41.527252 23781 raft_consensus.cc:374] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:41.527287 23781 raft_consensus.cc:729] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe06a14733cd48babfb611bd858b98cd, State: Initialized, Role: FOLLOWER
I20260619 02:49:41.527416 23781 consensus_queue.cc:260] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } }
I20260619 02:49:41.527671 23781 ts_tablet_manager.cc:1428] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:41.527586 23782 raft_consensus.cc:348] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } }
I20260619 02:49:41.527691 23621 heartbeater.cc:503] Master 127.21.164.254:45781 was elected leader, sending a full tablet report...
I20260619 02:49:41.527720 23782 raft_consensus.cc:374] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:41.527757 23782 raft_consensus.cc:729] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6cfe7b877e41f4bfb3e56adf8c33cd, State: Initialized, Role: FOLLOWER
I20260619 02:49:41.527864 23783 tablet_bootstrap.cc:492] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: Bootstrap starting.
I20260619 02:49:41.527874 23782 consensus_queue.cc:260] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } }
I20260619 02:49:41.528100 23782 ts_tablet_manager.cc:1428] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 02:49:41.528360 23761 heartbeater.cc:503] Master 127.21.164.254:45781 was elected leader, sending a full tablet report...
I20260619 02:49:41.528491 23783 tablet_bootstrap.cc:654] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:41.528791 23783 log.cc:824] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:41.529394 23783 tablet_bootstrap.cc:492] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: No bootstrap required, opened a new log
I20260619 02:49:41.529445 23783 ts_tablet_manager.cc:1397] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:41.530845 23783 raft_consensus.cc:348] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } }
I20260619 02:49:41.530965 23783 raft_consensus.cc:374] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:41.530985 23783 raft_consensus.cc:729] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96750281b69941fe9a00dab6e287bd6a, State: Initialized, Role: FOLLOWER
I20260619 02:49:41.531066 23783 consensus_queue.cc:260] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } }
I20260619 02:49:41.531288 23783 ts_tablet_manager.cc:1428] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:41.531353 23481 heartbeater.cc:503] Master 127.21.164.254:45781 was elected leader, sending a full tablet report...
W20260619 02:49:41.570281 23622 tablet.cc:2369] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:41.675359 23482 tablet.cc:2369] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:41.732316 23762 tablet.cc:2369] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:41.750111 23789 raft_consensus.cc:482] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:41.750231 23789 raft_consensus.cc:504] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } }
I20260619 02:49:41.750574 23789 leader_election.cc:288] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe06a14733cd48babfb611bd858b98cd (127.21.164.194:37971), 5c6cfe7b877e41f4bfb3e56adf8c33cd (127.21.164.195:46063)
I20260619 02:49:41.753533 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.750846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:56243 (local address 127.21.164.195:46063)
0619 02:49:41.750905 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.750907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.751010 (+ 103us) server_negotiation.cc:407] Connection header received
0619 02:49:41.751067 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.751068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.751086 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.751139 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:41.751286 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.751788 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.752854 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.753225 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.753291 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.753300 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.753430 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.753467 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:49:41.753533 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.750872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:40697 (local address 127.21.164.194:37971)
0619 02:49:41.750914 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.750916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.750977 (+ 61us) server_negotiation.cc:407] Connection header received
0619 02:49:41.751032 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.751033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.751043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.751098 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:41.751306 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.751807 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.752855 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.753218 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.753279 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.753288 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.753425 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.753461 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:49:41.753569 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.750754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:37971 (local address 127.21.164.193:40697)
0619 02:49:41.750900 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.750911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.750978 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.751108 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.751109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.751112 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:41.751205 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.751207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.751889 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.751890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.752776 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.752778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.753293 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.753294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.753361 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.753365 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.753411 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.753459 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":34,"thread_start_us":68,"threads_started":1}
I20260619 02:49:41.753576 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.750755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:46063 (local address 127.21.164.193:56243)
0619 02:49:41.750916 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.750927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.751012 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.751159 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.751160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.751162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:41.751230 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.751232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.751884 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.751885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.752778 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.752780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.753304 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.753305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.753369 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.753373 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.753420 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.753471 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":64,"spinlock_wait_cycles":17280,"thread_start_us":80,"threads_started":1}
I20260619 02:49:41.753885 23575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ba827374614ad485e2f0d6a9e4f0b9" candidate_uuid: "96750281b69941fe9a00dab6e287bd6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe06a14733cd48babfb611bd858b98cd" is_pre_election: true
I20260619 02:49:41.754047 23575 raft_consensus.cc:2454] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96750281b69941fe9a00dab6e287bd6a in term 0.
I20260619 02:49:41.754091 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ba827374614ad485e2f0d6a9e4f0b9" candidate_uuid: "96750281b69941fe9a00dab6e287bd6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" is_pre_election: true
I20260619 02:49:41.754218 23715 raft_consensus.cc:2454] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96750281b69941fe9a00dab6e287bd6a in term 0.
I20260619 02:49:41.754287 23367 leader_election.cc:302] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96750281b69941fe9a00dab6e287bd6a, fe06a14733cd48babfb611bd858b98cd; no voters:
I20260619 02:49:41.754406 23789 raft_consensus.cc:2781] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:41.754470 23789 raft_consensus.cc:482] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:41.754494 23789 raft_consensus.cc:3037] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:41.755160 23789 raft_consensus.cc:504] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } }
I20260619 02:49:41.755275 23789 leader_election.cc:288] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [CANDIDATE]: Term 1 election: Requested vote from peers fe06a14733cd48babfb611bd858b98cd (127.21.164.194:37971), 5c6cfe7b877e41f4bfb3e56adf8c33cd (127.21.164.195:46063)
I20260619 02:49:41.755496 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ba827374614ad485e2f0d6a9e4f0b9" candidate_uuid: "96750281b69941fe9a00dab6e287bd6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd"
I20260619 02:49:41.755492 23575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ba827374614ad485e2f0d6a9e4f0b9" candidate_uuid: "96750281b69941fe9a00dab6e287bd6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe06a14733cd48babfb611bd858b98cd"
I20260619 02:49:41.755576 23715 raft_consensus.cc:3037] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:41.755576 23575 raft_consensus.cc:3037] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:41.756408 23715 raft_consensus.cc:2454] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96750281b69941fe9a00dab6e287bd6a in term 1.
I20260619 02:49:41.756408 23575 raft_consensus.cc:2454] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96750281b69941fe9a00dab6e287bd6a in term 1.
I20260619 02:49:41.756589 23366 leader_election.cc:302] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6cfe7b877e41f4bfb3e56adf8c33cd, 96750281b69941fe9a00dab6e287bd6a; no voters:
I20260619 02:49:41.756699 23789 raft_consensus.cc:2781] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:41.756811 23789 raft_consensus.cc:686] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 1 LEADER]: Becoming Leader. State: Replica: 96750281b69941fe9a00dab6e287bd6a, State: Running, Role: LEADER
I20260619 02:49:41.756894 23789 consensus_queue.cc:237] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } }
I20260619 02:49:41.757665 23277 catalog_manager.cc:5495] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96750281b69941fe9a00dab6e287bd6a (127.21.164.193). New cstate: current_term: 1 leader_uuid: "96750281b69941fe9a00dab6e287bd6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } health_report { overall_health: HEALTHY } } }
I20260619 02:49:41.801011 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.798916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.21.164.193:37427)
0619 02:49:41.798964 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.798965 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.798972 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:49:41.799037 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.799038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.799055 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.799107 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:41.799208 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.799660 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.800380 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.800562 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.800611 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.800756 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:41.800845 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:41.800904 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.800923 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.800940 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:49:41.802690 23715 raft_consensus.cc:1261] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 1 FOLLOWER]: Refusing update from remote peer 96750281b69941fe9a00dab6e287bd6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:41.802704 23575 raft_consensus.cc:1261] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd [term 1 FOLLOWER]: Refusing update from remote peer 96750281b69941fe9a00dab6e287bd6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:41.802986 23792 consensus_queue.cc:1035] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:41.803122 23789 consensus_queue.cc:1035] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:41.805296 23795 mvcc.cc:205] Tried to move back new op lower bound from 7298405915861327872 to 7298405915676811264. Current Snapshot: MvccSnapshot[applied={T|T < 7298405915861327872}]
I20260619 02:49:41.805622 23797 mvcc.cc:205] Tried to move back new op lower bound from 7298405915861327872 to 7298405915676811264. Current Snapshot: MvccSnapshot[applied={T|T < 7298405915861327872}]
I20260619 02:49:41.808734 23277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:41.808831 23277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:41.809525 23277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32: Sending DeleteTablet for 3 replicas of tablet 56ba827374614ad485e2f0d6a9e4f0b9
I20260619 02:49:41.809847 23555 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ba827374614ad485e2f0d6a9e4f0b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:41 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:49396
I20260619 02:49:41.809870 23695 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ba827374614ad485e2f0d6a9e4f0b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:41 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:42440
I20260619 02:49:41.810083 23415 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ba827374614ad485e2f0d6a9e4f0b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:41 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:48682
I20260619 02:49:41.810143 23277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:49:41.810273 23806 tablet_replica.cc:330] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: stopping tablet replica
W20260619 02:49:41.810281 23277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:41.810360 23806 raft_consensus.cc:2229] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:41.810453 23806 raft_consensus.cc:2258] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:41.810596 23805 tablet_replica.cc:330] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: stopping tablet replica
I20260619 02:49:41.810691 23805 raft_consensus.cc:2229] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:41.810788 23805 raft_consensus.cc:2258] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:41.811545 23415 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5751f331194f529444cc466f88555d (DEFAULT_TABLE table=test-table [id=c1b8d129ffc2459ea1628d9e16328a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:41.811559 23695 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5751f331194f529444cc466f88555d (DEFAULT_TABLE table=test-table [id=c1b8d129ffc2459ea1628d9e16328a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:41.811666 23695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5751f331194f529444cc466f88555d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:41.811666 23415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5751f331194f529444cc466f88555d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:41.812788 23555 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5751f331194f529444cc466f88555d (DEFAULT_TABLE table=test-table [id=c1b8d129ffc2459ea1628d9e16328a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:41.812907 23555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5751f331194f529444cc466f88555d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:41.812981 23783 tablet_bootstrap.cc:492] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a: Bootstrap starting.
I20260619 02:49:41.813009 23782 tablet_bootstrap.cc:492] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Bootstrap starting.
I20260619 02:49:41.813411 23782 tablet_bootstrap.cc:654] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:41.813411 23783 tablet_bootstrap.cc:654] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:41.813994 23804 tablet_replica.cc:330] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: stopping tablet replica
I20260619 02:49:41.814162 23781 tablet_bootstrap.cc:492] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd: Bootstrap starting.
I20260619 02:49:41.814570 23806 ts_tablet_manager.cc:1905] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:41.814687 23781 tablet_bootstrap.cc:654] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:41.814098 23804 raft_consensus.cc:2229] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:41.814159 23782 tablet_bootstrap.cc:492] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd: No bootstrap required, opened a new log
I20260619 02:49:41.814975 23782 ts_tablet_manager.cc:1397] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:49:41.815142 23782 raft_consensus.cc:348] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.815222 23782 raft_consensus.cc:374] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:41.815243 23782 raft_consensus.cc:729] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6cfe7b877e41f4bfb3e56adf8c33cd, State: Initialized, Role: FOLLOWER
I20260619 02:49:41.815280 23782 consensus_queue.cc:260] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.815310 23781 tablet_bootstrap.cc:492] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd: No bootstrap required, opened a new log
I20260619 02:49:41.815341 23781 ts_tablet_manager.cc:1397] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:41.815361 23782 ts_tablet_manager.cc:1428] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.815467 23781 raft_consensus.cc:348] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.815536 23781 raft_consensus.cc:374] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:41.815554 23781 raft_consensus.cc:729] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe06a14733cd48babfb611bd858b98cd, State: Initialized, Role: FOLLOWER
I20260619 02:49:41.815590 23781 consensus_queue.cc:260] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.815667 23781 ts_tablet_manager.cc:1428] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.816089 23804 raft_consensus.cc:2258] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:41.816386 23806 ts_tablet_manager.cc:1918] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:41.816509 23806 log.cc:1196] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/wal/wals/56ba827374614ad485e2f0d6a9e4f0b9
I20260619 02:49:41.816532 23804 ts_tablet_manager.cc:1905] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:41.816859 23806 ts_tablet_manager.cc:1939] T 56ba827374614ad485e2f0d6a9e4f0b9 P 96750281b69941fe9a00dab6e287bd6a: Deleting consensus metadata
I20260619 02:49:41.817322 23263 catalog_manager.cc:4841] TS 96750281b69941fe9a00dab6e287bd6a (127.21.164.193:37427): tablet 56ba827374614ad485e2f0d6a9e4f0b9 (table test-table [id=c1293c6cf31e4952a16913ab1b85b862]) successfully deleted
I20260619 02:49:41.817919 23804 ts_tablet_manager.cc:1918] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:41.817987 23804 log.cc:1196] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/56ba827374614ad485e2f0d6a9e4f0b9
I20260619 02:49:41.818110 23783 tablet_bootstrap.cc:492] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a: No bootstrap required, opened a new log
I20260619 02:49:41.818181 23783 ts_tablet_manager.cc:1397] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 02:49:41.818210 23804 ts_tablet_manager.cc:1939] T 56ba827374614ad485e2f0d6a9e4f0b9 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Deleting consensus metadata
I20260619 02:49:41.818331 23783 raft_consensus.cc:348] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.818413 23783 raft_consensus.cc:374] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:41.818431 23783 raft_consensus.cc:729] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96750281b69941fe9a00dab6e287bd6a, State: Initialized, Role: FOLLOWER
I20260619 02:49:41.818483 23783 consensus_queue.cc:260] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.818570 23783 ts_tablet_manager.cc:1428] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.818601 23262 catalog_manager.cc:4841] TS 5c6cfe7b877e41f4bfb3e56adf8c33cd (127.21.164.195:46063): tablet 56ba827374614ad485e2f0d6a9e4f0b9 (table test-table [id=c1293c6cf31e4952a16913ab1b85b862]) successfully deleted
I20260619 02:49:41.818814 23805 ts_tablet_manager.cc:1905] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:41.819993 23805 ts_tablet_manager.cc:1918] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:41.820058 23805 log.cc:1196] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/56ba827374614ad485e2f0d6a9e4f0b9
I20260619 02:49:41.820250 23805 ts_tablet_manager.cc:1939] T 56ba827374614ad485e2f0d6a9e4f0b9 P fe06a14733cd48babfb611bd858b98cd: Deleting consensus metadata
I20260619 02:49:41.820683 23263 catalog_manager.cc:4841] TS fe06a14733cd48babfb611bd858b98cd (127.21.164.194:37971): tablet 56ba827374614ad485e2f0d6a9e4f0b9 (table test-table [id=c1293c6cf31e4952a16913ab1b85b862]) successfully deleted
I20260619 02:49:41.886438 23787 raft_consensus.cc:482] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:41.886574 23787 raft_consensus.cc:504] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.886911 23787 leader_election.cc:288] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96750281b69941fe9a00dab6e287bd6a (127.21.164.193:37427), fe06a14733cd48babfb611bd858b98cd (127.21.164.194:37971)
I20260619 02:49:41.890209 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.887088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:37427 (local address 127.21.164.195:47111)
0619 02:49:41.887148 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.887161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.887216 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.887412 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.887413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.887415 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:41.887478 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.887480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.888104 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.888105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.889220 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.889223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.889952 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.889952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.890031 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.890035 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.890086 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.890127 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 02:49:41.890208 23778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.887344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:37835 (local address 127.21.164.194:37971)
0619 02:49:41.887386 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.887388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.887397 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:41.887422 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.887423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.887431 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.887492 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:41.887620 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.888336 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.889526 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.889887 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.889947 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.889957 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.890114 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.890150 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:49:41.890209 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.887138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:47111 (local address 127.21.164.193:37427)
0619 02:49:41.887178 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.887179 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.887201 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:49:41.887280 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.887280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.887290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.887336 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:41.887556 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.888030 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.889313 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.889866 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.889948 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.889957 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.890100 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.890139 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:49:41.890399 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.887185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:37971 (local address 127.21.164.195:37835)
0619 02:49:41.887319 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.887327 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.887366 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.887498 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.887499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.887500 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:41.887556 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.887558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.888429 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.888430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.889456 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.889458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.889983 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.889984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.890042 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.890044 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.890282 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.890318 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260619 02:49:41.890632 23575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5751f331194f529444cc466f88555d" candidate_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe06a14733cd48babfb611bd858b98cd" is_pre_election: true
I20260619 02:49:41.890638 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5751f331194f529444cc466f88555d" candidate_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96750281b69941fe9a00dab6e287bd6a" is_pre_election: true
I20260619 02:49:41.890734 23575 raft_consensus.cc:2454] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c6cfe7b877e41f4bfb3e56adf8c33cd in term 0.
I20260619 02:49:41.890754 23435 raft_consensus.cc:2454] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c6cfe7b877e41f4bfb3e56adf8c33cd in term 0.
I20260619 02:49:41.890973 23647 leader_election.cc:302] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6cfe7b877e41f4bfb3e56adf8c33cd, 96750281b69941fe9a00dab6e287bd6a; no voters:
I20260619 02:49:41.891119 23787 raft_consensus.cc:2781] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:41.891163 23787 raft_consensus.cc:482] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:41.891202 23787 raft_consensus.cc:3037] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:41.891847 23787 raft_consensus.cc:504] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.891974 23787 leader_election.cc:288] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [CANDIDATE]: Term 1 election: Requested vote from peers 96750281b69941fe9a00dab6e287bd6a (127.21.164.193:37427), fe06a14733cd48babfb611bd858b98cd (127.21.164.194:37971)
I20260619 02:49:41.892133 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5751f331194f529444cc466f88555d" candidate_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96750281b69941fe9a00dab6e287bd6a"
I20260619 02:49:41.892202 23435 raft_consensus.cc:3037] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:41.892163 23575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5751f331194f529444cc466f88555d" candidate_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe06a14733cd48babfb611bd858b98cd"
I20260619 02:49:41.892234 23575 raft_consensus.cc:3037] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:41.892917 23435 raft_consensus.cc:2454] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c6cfe7b877e41f4bfb3e56adf8c33cd in term 1.
I20260619 02:49:41.893026 23575 raft_consensus.cc:2454] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c6cfe7b877e41f4bfb3e56adf8c33cd in term 1.
I20260619 02:49:41.893076 23647 leader_election.cc:302] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6cfe7b877e41f4bfb3e56adf8c33cd, 96750281b69941fe9a00dab6e287bd6a; no voters:
I20260619 02:49:41.893179 23787 raft_consensus.cc:2781] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:41.893280 23787 raft_consensus.cc:686] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 1 LEADER]: Becoming Leader. State: Replica: 5c6cfe7b877e41f4bfb3e56adf8c33cd, State: Running, Role: LEADER
I20260619 02:49:41.893342 23787 consensus_queue.cc:237] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.894090 23278 catalog_manager.cc:5495] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c6cfe7b877e41f4bfb3e56adf8c33cd (127.21.164.195). New cstate: current_term: 1 leader_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } health_report { overall_health: HEALTHY } } }
Jun 19 02:49:41 dist-test-slave-jt0t krb5kdc[23233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837380, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:41.928222 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.921422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.21.164.254:45781)
0619 02:49:41.921469 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:41.921471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:41.921591 (+ 120us) server_negotiation.cc:407] Connection header received
0619 02:49:41.921666 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:41.921666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:41.921675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:41.921722 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:41.922289 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.922861 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.923745 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.923922 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.923965 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.926161 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:41.926163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:41.926163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:41.926170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:41.927234 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.927552 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.927553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.927553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.927602 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:41.927752 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:41.927753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:41.927753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:41.927851 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:41.927906 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:41.928037 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 02:49:41.928115 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:49:41.928298 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:41.921326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45781 (local address 127.0.0.1:39300)
0619 02:49:41.921514 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:41.921525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:41.921604 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:41.921761 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:41.921762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:41.922158 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:41.922215 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.922217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.922946 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.922947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.923675 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:41.923677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:41.923979 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:41.923980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:41.924031 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:41.924573 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:41.924578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:41.926083 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:41.927300 (+ 1217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.927309 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.927312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.927489 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.927652 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:41.927652 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:41.927652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:41.927704 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:41.927901 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:41.927908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:41.927954 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:49:41.928010 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:49:41.928178 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":71,"thread_start_us":84,"threads_started":1}
I20260619 02:49:41.930182 23278 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:39300:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:49:41.930365 23278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:41.931614 23415 tablet_service.cc:1467] Processing CreateTablet for tablet 38080b64f73a45908c7b7917984178c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50b0fa7e2411485886e57f28c282ea1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:41.931641 23555 tablet_service.cc:1467] Processing CreateTablet for tablet 38080b64f73a45908c7b7917984178c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50b0fa7e2411485886e57f28c282ea1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:41.931766 23415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38080b64f73a45908c7b7917984178c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:41.931771 23555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38080b64f73a45908c7b7917984178c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:41.931717 23695 tablet_service.cc:1467] Processing CreateTablet for tablet 38080b64f73a45908c7b7917984178c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50b0fa7e2411485886e57f28c282ea1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:41.931859 23695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38080b64f73a45908c7b7917984178c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:41.932960 23782 tablet_bootstrap.cc:492] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Bootstrap starting.
I20260619 02:49:41.933156 23783 tablet_bootstrap.cc:492] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: Bootstrap starting.
I20260619 02:49:41.933377 23782 tablet_bootstrap.cc:654] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:41.933418 23783 tablet_bootstrap.cc:654] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:41.933612 23781 tablet_bootstrap.cc:492] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd: Bootstrap starting.
I20260619 02:49:41.933915 23781 tablet_bootstrap.cc:654] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:41.934299 23783 tablet_bootstrap.cc:492] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: No bootstrap required, opened a new log
I20260619 02:49:41.934341 23783 ts_tablet_manager.cc:1397] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:41.934451 23781 tablet_bootstrap.cc:492] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd: No bootstrap required, opened a new log
I20260619 02:49:41.934442 23783 raft_consensus.cc:348] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.934497 23783 raft_consensus.cc:374] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:41.934495 23781 ts_tablet_manager.cc:1397] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:41.934512 23783 raft_consensus.cc:729] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96750281b69941fe9a00dab6e287bd6a, State: Initialized, Role: FOLLOWER
I20260619 02:49:41.934553 23783 consensus_queue.cc:260] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.934640 23783 ts_tablet_manager.cc:1428] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.934648 23781 raft_consensus.cc:348] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.934662 23789 tablet_replica.cc:439] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } }
I20260619 02:49:41.934708 23781 raft_consensus.cc:374] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:41.934725 23781 raft_consensus.cc:729] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe06a14733cd48babfb611bd858b98cd, State: Initialized, Role: FOLLOWER
I20260619 02:49:41.934726 23789 tablet_replica.cc:442] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:41.934775 23781 consensus_queue.cc:260] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.934865 23781 ts_tablet_manager.cc:1428] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.934921 23786 tablet_replica.cc:439] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } }
I20260619 02:49:41.934996 23786 tablet_replica.cc:442] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:41.935015 23782 tablet_bootstrap.cc:492] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: No bootstrap required, opened a new log
I20260619 02:49:41.935056 23782 ts_tablet_manager.cc:1397] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:49:41.935184 23782 raft_consensus.cc:348] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.935256 23782 raft_consensus.cc:374] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:41.935281 23782 raft_consensus.cc:729] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6cfe7b877e41f4bfb3e56adf8c33cd, State: Initialized, Role: FOLLOWER
I20260619 02:49:41.935318 23782 consensus_queue.cc:260] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:41.935410 23782 ts_tablet_manager.cc:1428] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:41.935458 23787 tablet_replica.cc:439] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } }
I20260619 02:49:41.935545 23787 tablet_replica.cc:442] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:42.074265 23789 raft_consensus.cc:482] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:42.074378 23789 raft_consensus.cc:504] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:42.074515 23789 leader_election.cc:288] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe06a14733cd48babfb611bd858b98cd (127.21.164.194:37971), 5c6cfe7b877e41f4bfb3e56adf8c33cd (127.21.164.195:46063)
I20260619 02:49:42.074790 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38080b64f73a45908c7b7917984178c7" candidate_uuid: "96750281b69941fe9a00dab6e287bd6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" is_pre_election: true
I20260619 02:49:42.074782 23575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38080b64f73a45908c7b7917984178c7" candidate_uuid: "96750281b69941fe9a00dab6e287bd6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe06a14733cd48babfb611bd858b98cd" is_pre_election: true
I20260619 02:49:42.074895 23575 raft_consensus.cc:2454] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96750281b69941fe9a00dab6e287bd6a in term 0.
I20260619 02:49:42.074895 23715 raft_consensus.cc:2454] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96750281b69941fe9a00dab6e287bd6a in term 0.
I20260619 02:49:42.075040 23367 leader_election.cc:302] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96750281b69941fe9a00dab6e287bd6a, fe06a14733cd48babfb611bd858b98cd; no voters:
I20260619 02:49:42.075147 23789 raft_consensus.cc:2781] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:42.075184 23789 raft_consensus.cc:482] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:42.075197 23789 raft_consensus.cc:3037] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:42.075893 23789 raft_consensus.cc:504] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:42.076018 23789 leader_election.cc:288] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [CANDIDATE]: Term 1 election: Requested vote from peers fe06a14733cd48babfb611bd858b98cd (127.21.164.194:37971), 5c6cfe7b877e41f4bfb3e56adf8c33cd (127.21.164.195:46063)
I20260619 02:49:42.076188 23575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38080b64f73a45908c7b7917984178c7" candidate_uuid: "96750281b69941fe9a00dab6e287bd6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe06a14733cd48babfb611bd858b98cd"
I20260619 02:49:42.076205 23715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38080b64f73a45908c7b7917984178c7" candidate_uuid: "96750281b69941fe9a00dab6e287bd6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd"
I20260619 02:49:42.076269 23575 raft_consensus.cc:3037] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:42.076269 23715 raft_consensus.cc:3037] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:42.077134 23575 raft_consensus.cc:2454] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96750281b69941fe9a00dab6e287bd6a in term 1.
I20260619 02:49:42.077319 23367 leader_election.cc:302] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96750281b69941fe9a00dab6e287bd6a, fe06a14733cd48babfb611bd858b98cd; no voters:
I20260619 02:49:42.077425 23789 raft_consensus.cc:2781] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:42.077507 23789 raft_consensus.cc:686] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [term 1 LEADER]: Becoming Leader. State: Replica: 96750281b69941fe9a00dab6e287bd6a, State: Running, Role: LEADER
I20260619 02:49:42.077571 23715 raft_consensus.cc:2454] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96750281b69941fe9a00dab6e287bd6a in term 1.
I20260619 02:49:42.077562 23789 consensus_queue.cc:237] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } }
I20260619 02:49:42.077824 23792 tablet_replica.cc:439] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: TxnStatusTablet state changed. Reason: New leader 96750281b69941fe9a00dab6e287bd6a. Latest consensus state: current_term: 1 leader_uuid: "96750281b69941fe9a00dab6e287bd6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } }
I20260619 02:49:42.077910 23792 tablet_replica.cc:442] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:42.078088 23823 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:49:42.078197 23823 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:49:42.078218 23277 catalog_manager.cc:5495] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96750281b69941fe9a00dab6e287bd6a (127.21.164.193). New cstate: current_term: 1 leader_uuid: "96750281b69941fe9a00dab6e287bd6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:42.116899 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.114380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:37427 (local address 127.0.0.1:48704)
0619 02:49:42.114449 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:42.114462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:42.114508 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:42.114648 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:42.114648 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:42.114649 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:42.114695 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:42.114697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.115577 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.115578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:42.116148 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:42.116149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.116541 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.116541 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:42.116585 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.116601 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:42.116757 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:42.116761 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:42.116797 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 02:49:42.116836 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 02:49:42.116909 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.114467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.21.164.193:37427)
0619 02:49:42.114524 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:42.114526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:42.114537 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:42.114565 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:42.114566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:42.114577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:42.114643 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:42.114743 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.115489 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.116213 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.116478 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.116545 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.116675 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:42.116709 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:42.116750 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:42.116801 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 02:49:42.116819 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:42.118002 23715 raft_consensus.cc:1261] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 1 FOLLOWER]: Refusing update from remote peer 96750281b69941fe9a00dab6e287bd6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:42.118212 23575 raft_consensus.cc:1261] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd [term 1 FOLLOWER]: Refusing update from remote peer 96750281b69941fe9a00dab6e287bd6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:42.118247 23792 consensus_queue.cc:1035] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:42.118427 23792 consensus_queue.cc:1035] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:42.118749 23787 tablet_replica.cc:439] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: TxnStatusTablet state changed. Reason: New leader 96750281b69941fe9a00dab6e287bd6a. Latest consensus state: current_term: 1 leader_uuid: "96750281b69941fe9a00dab6e287bd6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } }
I20260619 02:49:42.118834 23787 tablet_replica.cc:442] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:42.118870 23786 tablet_replica.cc:439] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd: TxnStatusTablet state changed. Reason: New leader 96750281b69941fe9a00dab6e287bd6a. Latest consensus state: current_term: 1 leader_uuid: "96750281b69941fe9a00dab6e287bd6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } }
I20260619 02:49:42.118939 23786 tablet_replica.cc:442] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:42.119529 23789 tablet_replica.cc:439] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96750281b69941fe9a00dab6e287bd6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } }
I20260619 02:49:42.119604 23789 tablet_replica.cc:442] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:42.119673 23796 mvcc.cc:205] Tried to move back new op lower bound from 7298405917153570816 to 7298405916990078976. Current Snapshot: MvccSnapshot[applied={T|T < 7298405917153570816}]
I20260619 02:49:42.119884 23786 tablet_replica.cc:439] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96750281b69941fe9a00dab6e287bd6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } }
I20260619 02:49:42.119964 23786 tablet_replica.cc:442] T 38080b64f73a45908c7b7917984178c7 P fe06a14733cd48babfb611bd858b98cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:42.120729 23789 tablet_replica.cc:439] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96750281b69941fe9a00dab6e287bd6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } }
I20260619 02:49:42.120810 23789 tablet_replica.cc:442] T 38080b64f73a45908c7b7917984178c7 P 96750281b69941fe9a00dab6e287bd6a: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:42.121081 23787 tablet_replica.cc:439] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96750281b69941fe9a00dab6e287bd6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 } } peers { permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 } } peers { permanent_uuid: "5c6cfe7b877e41f4bfb3e56adf8c33cd" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46063 } } }
I20260619 02:49:42.121162 23787 tablet_replica.cc:442] T 38080b64f73a45908c7b7917984178c7 P 5c6cfe7b877e41f4bfb3e56adf8c33cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:42.127660 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.125058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.21.164.195:46063)
0619 02:49:42.125096 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:42.125098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:42.125139 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:49:42.125217 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:42.125217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:42.125239 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:42.125312 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:42.125427 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.125906 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.126944 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.127148 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.127208 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.127350 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:42.127456 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:42.127507 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:42.127557 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 02:49:42.127577 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:49:42.132643 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.130073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:41393 (local address 127.21.164.193:37427)
0619 02:49:42.130127 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:42.130130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:42.130195 (+ 65us) server_negotiation.cc:407] Connection header received
0619 02:49:42.130299 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:42.130300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:42.130309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:42.130360 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:42.130459 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.131163 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.131920 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.132232 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.132296 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.132429 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:42.132462 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:42.132503 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:42.132545 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 02:49:42.132564 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:49:42.132680 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.129971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:37427 (local address 127.21.164.195:41393)
0619 02:49:42.130143 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:42.130153 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:42.130205 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:42.130367 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:42.130368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:42.130369 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:42.130415 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:42.130416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.131239 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.131239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:42.131845 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:42.131847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.132303 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.132304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:42.132350 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.132362 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:42.132511 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:42.132512 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:49:42.132541 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 02:49:42.132583 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260619 02:49:42.134838 23575 raft_consensus.cc:1261] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd [term 1 FOLLOWER]: Refusing update from remote peer 5c6cfe7b877e41f4bfb3e56adf8c33cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:42.134893 23435 raft_consensus.cc:1261] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a [term 1 FOLLOWER]: Refusing update from remote peer 5c6cfe7b877e41f4bfb3e56adf8c33cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:42.135138 23811 consensus_queue.cc:1035] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "96750281b69941fe9a00dab6e287bd6a" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:42.135334 23787 consensus_queue.cc:1035] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe06a14733cd48babfb611bd858b98cd" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:42.143671 23835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.141180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:46063 (local address 127.21.164.193:50495)
0619 02:49:42.141331 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:42.141340 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:42.141395 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:42.141509 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:42.141510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:42.141512 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:42.141556 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:42.141558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.142126 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.142127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:42.142879 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:42.142881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.143340 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.143341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:42.143383 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.143396 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:42.143550 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:42.143551 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:49:42.143579 (+ 28us) client_negotiation.cc:240] Negotiation successful
0619 02:49:42.143612 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260619 02:49:42.143676 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.141245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:50495 (local address 127.21.164.195:46063)
0619 02:49:42.141294 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:42.141296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:42.141394 (+ 98us) server_negotiation.cc:407] Connection header received
0619 02:49:42.141448 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:42.141449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:42.141460 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:42.141499 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:42.141596 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.142063 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.142975 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.143270 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.143342 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.143465 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:42.143499 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:42.143551 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:42.143583 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 02:49:42.143605 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:49:42.149976 23277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:42.150087 23277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:42.150616 23277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8a689a67f1d46faa5eb1a9617c51f32: Sending DeleteTablet for 3 replicas of tablet 6a5751f331194f529444cc466f88555d
I20260619 02:49:42.150878 23555 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a5751f331194f529444cc466f88555d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:42 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:49396
I20260619 02:49:42.150969 23805 tablet_replica.cc:330] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd: stopping tablet replica
I20260619 02:49:42.151010 23805 raft_consensus.cc:2229] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:42.151038 23805 raft_consensus.cc:2258] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:42.151021 23695 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a5751f331194f529444cc466f88555d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:42 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:42440
I20260619 02:49:42.151106 23804 tablet_replica.cc:330] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd: stopping tablet replica
I20260619 02:49:42.151189 23804 raft_consensus.cc:2229] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:42.151273 23804 raft_consensus.cc:2258] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:42.151578 23804 ts_tablet_manager.cc:1905] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:42.151638 23805 ts_tablet_manager.cc:1905] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:42.153010 23805 ts_tablet_manager.cc:1918] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:42.153066 23805 log.cc:1196] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/6a5751f331194f529444cc466f88555d
I20260619 02:49:42.153204 23415 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a5751f331194f529444cc466f88555d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:42 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:48682
I20260619 02:49:42.153267 23805 ts_tablet_manager.cc:1939] T 6a5751f331194f529444cc466f88555d P fe06a14733cd48babfb611bd858b98cd: Deleting consensus metadata
I20260619 02:49:42.153291 23806 tablet_replica.cc:330] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a: stopping tablet replica
I20260619 02:49:42.153286 23804 ts_tablet_manager.cc:1918] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:42.153337 23804 log.cc:1196] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/6a5751f331194f529444cc466f88555d
I20260619 02:49:42.153335 23806 raft_consensus.cc:2229] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:42.153368 23806 raft_consensus.cc:2258] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:42.153545 23263 catalog_manager.cc:4841] TS fe06a14733cd48babfb611bd858b98cd (127.21.164.194:37971): tablet 6a5751f331194f529444cc466f88555d (table test-table [id=c1b8d129ffc2459ea1628d9e16328a63]) successfully deleted
I20260619 02:49:42.153592 23804 ts_tablet_manager.cc:1939] T 6a5751f331194f529444cc466f88555d P 5c6cfe7b877e41f4bfb3e56adf8c33cd: Deleting consensus metadata
I20260619 02:49:42.153924 23262 catalog_manager.cc:4841] TS 5c6cfe7b877e41f4bfb3e56adf8c33cd (127.21.164.195:46063): tablet 6a5751f331194f529444cc466f88555d (table test-table [id=c1b8d129ffc2459ea1628d9e16328a63]) successfully deleted
I20260619 02:49:42.153959 23806 ts_tablet_manager.cc:1905] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:42.155433 23806 ts_tablet_manager.cc:1918] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:42.155478 23806 log.cc:1196] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837375929108-22163-0/minicluster-data/ts-0/wal/wals/6a5751f331194f529444cc466f88555d
I20260619 02:49:42.155648 23806 ts_tablet_manager.cc:1939] T 6a5751f331194f529444cc466f88555d P 96750281b69941fe9a00dab6e287bd6a: Deleting consensus metadata
I20260619 02:49:42.156091 23263 catalog_manager.cc:4841] TS 96750281b69941fe9a00dab6e287bd6a (127.21.164.193:37427): tablet 6a5751f331194f529444cc466f88555d (table test-table [id=c1b8d129ffc2459ea1628d9e16328a63]) successfully deleted
Jun 19 02:49:42 dist-test-slave-jt0t krb5kdc[23233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:49:42.162429 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.153906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.21.164.193:37427)
0619 02:49:42.153945 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:42.153947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:42.154208 (+ 261us) server_negotiation.cc:407] Connection header received
0619 02:49:42.154292 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:42.154293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:42.154301 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:42.154327 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:42.155062 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.155765 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.156592 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.156846 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.156900 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.159581 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:42.159589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:42.159589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:42.159596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:42.161224 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:42.161519 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:42.161522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:42.161523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:42.161578 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:42.161764 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:42.161765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:42.161765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:42.161933 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:42.162023 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:42.162187 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 02:49:42.162266 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 02:49:42.162635 23475 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48710
I20260619 02:49:42.168926 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:42.163440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39306 (local address 127.21.164.254:45781)
0619 02:49:42.163486 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:42.163488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:42.163643 (+ 155us) server_negotiation.cc:407] Connection header received
0619 02:49:42.163698 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:42.163699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:42.163710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:42.163753 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:42.164217 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.164773 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.165423 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:42.165623 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:42.165678 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:42.166690 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:42.166692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:42.166692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:42.166699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:42.167925 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:42.168216 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:42.168217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:42.168217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:42.168264 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:42.168423 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:42.168423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:42.168424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:42.168545 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:42.168630 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:42.168757 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 02:49:42.168830 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 02:49:42.169248 23277 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39306
I20260619 02:49:42.169847 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 23350
I20260619 02:49:42.177158 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 23490
I20260619 02:49:42.182535 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 23630
I20260619 02:49:42.186704 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 23246
2026-06-19T02:49:42Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2393 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:42 dist-test-slave-jt0t krb5kdc[23845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:42 dist-test-slave-jt0t krb5kdc[23845](info): set up 2 sockets
Jun 19 02:49:42 dist-test-slave-jt0t krb5kdc[23845](info): commencing operation
krb5kdc: starting...
W20260619 02:49:43.140108 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:49:43 dist-test-slave-jt0t krb5kdc[23845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:49:43.252351 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42173
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:42173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:43.333056 23858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:43.333292 23858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:43.333323 23858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:43.334533 23858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:43.334576 23858 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:49:43.334592 23858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:43.334609 23858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:43.334625 23858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:43.334641 23858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:43.336185 23858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:42173
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:43.336452 23858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:43.336689 23858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:43.341377 23866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.341408 23863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.341406 23864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.341583 23858 server_base.cc:1034] running on GCE node
I20260619 02:49:43.341943 23858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:43.342244 23858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:43.343390 23858 hybrid_clock.cc:648] HybridClock initialized: now 1781837383343374 us; error 37 us; skew 500 ppm
Jun 19 02:49:43 dist-test-slave-jt0t krb5kdc[23845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:43.396525 23858 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:49:43.397087 23858 webserver.cc:460] Webserver started at http://127.21.164.254:39797/ using document root <none> and password file <none>
I20260619 02:49:43.397300 23858 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.397351 23858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.397477 23858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:43.398542 23858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "486e66c8d95148bf973aa60caa49ef1f"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "1e53947dc0bb4a0b28000b35c415509d"
server_key_iv: "0bed5b6e98b4aed6b403f50769ee22ad"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.398885 23858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "486e66c8d95148bf973aa60caa49ef1f"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "1e53947dc0bb4a0b28000b35c415509d"
server_key_iv: "0bed5b6e98b4aed6b403f50769ee22ad"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.400202 23858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:49:43.400965 23873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.401113 23858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:43.401201 23858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "486e66c8d95148bf973aa60caa49ef1f"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "1e53947dc0bb4a0b28000b35c415509d"
server_key_iv: "0bed5b6e98b4aed6b403f50769ee22ad"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.401261 23858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.415266 23858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:43.417505 23858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:43.417649 23858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:43.422017 23858 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:42173
I20260619 02:49:43.422132 23935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:42173 every 8 connection(s)
I20260619 02:49:43.422417 23858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:43.423084 23936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:43.425405 23936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f: Bootstrap starting.
I20260619 02:49:43.425998 23936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:43.426260 23936 log.cc:824] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:43.427094 23936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f: No bootstrap required, opened a new log
I20260619 02:49:43.427902 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 23858
I20260619 02:49:43.427991 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:43.428140 22163 external_mini_cluster.cc:1428] Setting key 3479be57ea916021022a211fee3f7ab7
I20260619 02:49:43.429227 23936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486e66c8d95148bf973aa60caa49ef1f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42173 } }
I20260619 02:49:43.429383 23936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:43.429406 23936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486e66c8d95148bf973aa60caa49ef1f, State: Initialized, Role: FOLLOWER
I20260619 02:49:43.429523 23936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486e66c8d95148bf973aa60caa49ef1f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42173 } }
I20260619 02:49:43.429589 23936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:43.429612 23936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:43.429687 23936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:43.430212 23936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486e66c8d95148bf973aa60caa49ef1f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42173 } }
I20260619 02:49:43.430306 23936 leader_election.cc:302] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486e66c8d95148bf973aa60caa49ef1f; no voters:
I20260619 02:49:43.430461 23936 leader_election.cc:288] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:43.430500 23941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:43.430642 23941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [term 1 LEADER]: Becoming Leader. State: Replica: 486e66c8d95148bf973aa60caa49ef1f, State: Running, Role: LEADER
I20260619 02:49:43.430725 23936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:43.430780 23941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486e66c8d95148bf973aa60caa49ef1f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42173 } }
I20260619 02:49:43.431172 23941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486e66c8d95148bf973aa60caa49ef1f. Latest consensus state: current_term: 1 leader_uuid: "486e66c8d95148bf973aa60caa49ef1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486e66c8d95148bf973aa60caa49ef1f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42173 } } }
I20260619 02:49:43.431233 23943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "486e66c8d95148bf973aa60caa49ef1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486e66c8d95148bf973aa60caa49ef1f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42173 } } }
I20260619 02:49:43.431365 23941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:43.431375 23943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:43.431574 23950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:43.432121 23950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 02:49:43 dist-test-slave-jt0t krb5kdc[23845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:43.436468 23950 catalog_manager.cc:1347] Generated new cluster ID: 8a42a31d5feb4cd3b47f088a52df2bad
I20260619 02:49:43.436518 23950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:43.439997 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.428684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44432 (local address 127.21.164.254:42173)
0619 02:49:43.429296 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.429305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.429331 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:49:43.429464 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.429472 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.429532 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.429677 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.430276 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.431582 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.432601 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.433153 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.433244 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.435816 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.435821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.435821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.435834 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.437938 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.438340 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.438343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.438343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.438484 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.438689 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.438690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.438690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.439018 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.439183 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.439627 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.439773 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":523,"threads_started":1}
I20260619 02:49:43.446602 23950 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:43.447118 23950 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:43.451510 23950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f: Generated new TSK 0
I20260619 02:49:43.451681 23950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:49:43.481396 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42173
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:43.577055 23962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:43.577268 23962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:43.577293 23962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:43.578449 23962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:43.578485 23962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:43.578505 23962 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:49:43.578516 23962 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:43.578531 23962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:49:43.579927 23962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.164.254:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:43.580107 23962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:43.580288 23962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:43.584978 23970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.585063 23962 server_base.cc:1034] running on GCE node
W20260619 02:49:43.584985 23968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.584996 23967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.585429 23962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:43.585654 23962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:43.586787 23962 hybrid_clock.cc:648] HybridClock initialized: now 1781837383586775 us; error 28 us; skew 500 ppm
Jun 19 02:49:43 dist-test-slave-jt0t krb5kdc[23845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:43.589195 23962 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:49:43.589618 23962 webserver.cc:460] Webserver started at http://127.21.164.193:43061/ using document root <none> and password file <none>
I20260619 02:49:43.589813 23962 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.589859 23962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.589979 23962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:43.590992 23962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "1d4a5220ff004058a1e0d067c4db4b7e"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "e45d3f4ba39362c05a839c72f191a708"
server_key_iv: "87ae785a5e93d5aaa329cff7f3195f5b"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.591347 23962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "1d4a5220ff004058a1e0d067c4db4b7e"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "e45d3f4ba39362c05a839c72f191a708"
server_key_iv: "87ae785a5e93d5aaa329cff7f3195f5b"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.592551 23962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:43.593245 23977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.593393 23962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.593472 23962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "1d4a5220ff004058a1e0d067c4db4b7e"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "e45d3f4ba39362c05a839c72f191a708"
server_key_iv: "87ae785a5e93d5aaa329cff7f3195f5b"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.593531 23962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.606705 23962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:43.608737 23962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:43.608881 23962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:43.609400 23962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:43.609441 23962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.609470 23962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:43.609488 23962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.615589 23962 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:44941
I20260619 02:49:43.615669 24092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:44941 every 8 connection(s)
I20260619 02:49:43.615947 23962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:49:43 dist-test-slave-jt0t krb5kdc[23845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:43.616786 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 23962
I20260619 02:49:43.616873 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:43.617033 22163 external_mini_cluster.cc:1428] Setting key ce77156189b948ea70a9b658dbbb8d22
I20260619 02:49:43.619364 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.609837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:55365 (local address 127.21.164.254:42173)
0619 02:49:43.609880 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.609881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.610110 (+ 229us) server_negotiation.cc:407] Connection header received
0619 02:49:43.610591 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.610592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.610609 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.610674 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.611559 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.612343 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.613857 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.614152 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.614234 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.616583 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.616585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.616585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.616595 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.617989 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.618355 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.618356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.618356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.618417 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.618677 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.618678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.618678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.618811 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.618868 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.619179 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.619297 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:49:43.619554 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.609851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42173 (local address 127.21.164.193:55365)
0619 02:49:43.610002 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.610039 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.610501 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.610763 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.610767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.611097 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.611451 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.611458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.612447 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.612448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.613754 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.613757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.614276 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.614277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.614390 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.614992 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.615001 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.616495 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.618066 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.618072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.618075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.618289 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.618469 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.618470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.618470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.618603 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.618870 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.618875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.619041 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.619152 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.619321 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260619 02:49:43.620007 24093 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42173
I20260619 02:49:43.620092 24093 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:43.620254 24093 heartbeater.cc:511] Master 127.21.164.254:42173 requested a full tablet report, sending...
I20260619 02:49:43.620780 23889 ts_manager.cc:194] Registered new tserver with Master: 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193:44941)
I20260619 02:49:43.621483 23889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:55365
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
I20260619 02:49:43.627144 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.621102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:50303 (local address 127.21.164.254:42173)
0619 02:49:43.621139 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.621140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.621267 (+ 127us) server_negotiation.cc:407] Connection header received
0619 02:49:43.621354 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.621355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.621365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.621393 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.621795 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.622536 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.623551 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.623745 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.623800 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.624822 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.624823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.624824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.624830 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.626147 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.626534 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.626535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.626536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.626596 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.626763 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.626764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.626764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.626866 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.626912 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.627011 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.627076 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 02:49:43.627175 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.621041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42173 (local address 127.21.164.193:50303)
0619 02:49:43.621198 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.621208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.621290 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.621451 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.621452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.621648 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.621710 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.621713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.622646 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.622647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.623457 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.623460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.623819 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.623820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.623896 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.624333 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.624338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.624728 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.626250 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.626252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.626252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.626450 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.626664 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.626664 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.626664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.626724 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.626908 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.626909 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.626949 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.626995 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.627080 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:49:43.642975 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42173
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:43.723595 24102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:43.723798 24102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:43.723824 24102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:43.724963 24102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:43.724998 24102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:43.725016 24102 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:49:43.725028 24102 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:43.725042 24102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:49:43.726429 24102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.164.254:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:43.726612 24102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:43.726784 24102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:43.731442 24107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.731442 24108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.731436 24110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.731756 24102 server_base.cc:1034] running on GCE node
I20260619 02:49:43.731926 24102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:43.732149 24102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:43.733299 24102 hybrid_clock.cc:648] HybridClock initialized: now 1781837383733284 us; error 28 us; skew 500 ppm
Jun 19 02:49:43 dist-test-slave-jt0t krb5kdc[23845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:43.735672 24102 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:49:43.736111 24102 webserver.cc:460] Webserver started at http://127.21.164.194:35923/ using document root <none> and password file <none>
I20260619 02:49:43.736310 24102 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.736358 24102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.736490 24102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:43.737617 24102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "75a99e4eb38b443e8ec6770ab3425739"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "e5544b0612addc808bdb1a05ea97062f"
server_key_iv: "4a065e9db26870d584ff3b7298613e96"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.737972 24102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "75a99e4eb38b443e8ec6770ab3425739"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "e5544b0612addc808bdb1a05ea97062f"
server_key_iv: "4a065e9db26870d584ff3b7298613e96"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.739416 24102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:43.740127 24117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.740305 24102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:43.740391 24102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "75a99e4eb38b443e8ec6770ab3425739"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "e5544b0612addc808bdb1a05ea97062f"
server_key_iv: "4a065e9db26870d584ff3b7298613e96"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.740468 24102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.746739 24102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:43.748428 24102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:43.748557 24102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:43.749054 24102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:43.749097 24102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.749128 24102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:43.749145 24102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.755164 24102 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:45587
I20260619 02:49:43.755282 24232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:45587 every 8 connection(s)
I20260619 02:49:43.755563 24102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:49:43 dist-test-slave-jt0t krb5kdc[23845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:43.757382 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 24102
I20260619 02:49:43.757468 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:43.757647 22163 external_mini_cluster.cc:1428] Setting key cf7e612c3887f6aaa1f1302fc0bd2c05
I20260619 02:49:43.758705 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.749502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:51755 (local address 127.21.164.254:42173)
0619 02:49:43.749570 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.749573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.749735 (+ 162us) server_negotiation.cc:407] Connection header received
0619 02:49:43.750203 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.750204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.750216 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.750303 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.751088 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.751869 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.753342 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.753628 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.753708 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.756210 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.756211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.756212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.756218 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.757401 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.757783 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.757784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.757784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.757844 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.758097 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.758097 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.758098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.758211 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.758271 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.758533 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.758618 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:49:43.758877 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.749456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42173 (local address 127.21.164.194:51755)
0619 02:49:43.749607 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.749646 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.750112 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.750368 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.750372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.750703 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.750981 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.750987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.751976 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.751977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.753244 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.753248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.753708 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.753709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.753808 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.754477 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.754486 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.756124 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.757476 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.757482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.757484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.757724 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.757897 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.757898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.757898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.758037 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.758262 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.758268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.758414 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.758506 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.758654 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260619 02:49:43.759270 24233 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42173
I20260619 02:49:43.759357 24233 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:43.759586 24233 heartbeater.cc:511] Master 127.21.164.254:42173 requested a full tablet report, sending...
I20260619 02:49:43.759974 23889 ts_manager.cc:194] Registered new tserver with Master: 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587)
I20260619 02:49:43.760440 23889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:51755
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
I20260619 02:49:43.765264 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.760228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42173 (local address 127.21.164.194:43439)
0619 02:49:43.760358 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.760367 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.760421 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.760570 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.760571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.760762 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.760815 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.760816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.761447 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.761448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.761994 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.761996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.762316 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.762318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.762389 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.762761 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.762766 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.763026 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.764339 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.764340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.764340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.764500 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.764680 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.764680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.764681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.764739 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.764994 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.764995 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.765040 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.765103 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.765190 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260619 02:49:43.765311 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.760286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:43439 (local address 127.21.164.254:42173)
0619 02:49:43.760320 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.760321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.760426 (+ 105us) server_negotiation.cc:407] Connection header received
0619 02:49:43.760465 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.760466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.760475 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.760529 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.760881 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.761355 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.762063 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.762261 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.762312 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.763103 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.763104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.763104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.763111 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.764269 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.764558 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.764559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.764560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.764611 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.764790 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.764791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.764791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.764937 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.764993 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.765142 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.765219 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:49:43.783977 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42173
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:43.862646 24242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:43.862854 24242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:43.862879 24242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:43.863976 24242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:43.864012 24242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:43.864030 24242 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:49:43.864041 24242 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:49:43.864056 24242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:49:43.865454 24242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.164.254:42173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:43.865691 24242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:43.865899 24242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:43.870568 24248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.870628 24247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:43.870589 24250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:43.870743 24242 server_base.cc:1034] running on GCE node
I20260619 02:49:43.870990 24242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:43.871236 24242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:43.872382 24242 hybrid_clock.cc:648] HybridClock initialized: now 1781837383872365 us; error 31 us; skew 500 ppm
Jun 19 02:49:43 dist-test-slave-jt0t krb5kdc[23845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:43.874782 24242 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:49:43.875203 24242 webserver.cc:460] Webserver started at http://127.21.164.195:36649/ using document root <none> and password file <none>
I20260619 02:49:43.875432 24242 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:43.875481 24242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:43.875607 24242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:43.876682 24242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "74c580b8d4554aa38b61570f591243e9"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "2562f83953e1744b927b01a92a2ee6ef"
server_key_iv: "44183254c368b90be66da781cb8abb39"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.877033 24242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "74c580b8d4554aa38b61570f591243e9"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "2562f83953e1744b927b01a92a2ee6ef"
server_key_iv: "44183254c368b90be66da781cb8abb39"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.878275 24242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:43.878962 24257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.879104 24242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:49:43.879184 24242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "74c580b8d4554aa38b61570f591243e9"
format_stamp: "Formatted at 2026-06-19 02:49:43 on dist-test-slave-jt0t"
server_key: "2562f83953e1744b927b01a92a2ee6ef"
server_key_iv: "44183254c368b90be66da781cb8abb39"
server_key_version: "encryptionkey@0"
I20260619 02:49:43.879244 24242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:43.900873 24242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:43.902631 24242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:43.902738 24242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:43.903185 24242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:43.903223 24242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:43.903247 24242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:43.903258 24242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:49:43 dist-test-slave-jt0t krb5kdc[23845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:43.909775 24242 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:33457
I20260619 02:49:43.909848 24372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:33457 every 8 connection(s)
I20260619 02:49:43.910125 24242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:43.912364 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.903684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:35327 (local address 127.21.164.254:42173)
0619 02:49:43.903731 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.903734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.903993 (+ 259us) server_negotiation.cc:407] Connection header received
0619 02:49:43.904496 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.904498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.904510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.904560 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.905306 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.905907 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.906890 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.907093 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.907147 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.909731 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.909733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.909734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.909742 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.911114 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.911495 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.911496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.911497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.911554 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.911758 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.911759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.911759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.911882 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.911942 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.912194 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.912274 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:49:43.912539 24266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.903696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42173 (local address 127.21.164.195:35327)
0619 02:49:43.903864 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.903910 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.904400 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.904631 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.904635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.905009 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.905214 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.905220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.905994 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.905995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.906804 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.906806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.907157 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.907158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.907306 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.907797 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.907808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.909642 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.911200 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.911205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.911207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.911428 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.911605 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.911606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.911606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.911706 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.911937 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.911942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.912076 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.912165 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.912317 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":98,"threads_started":1}
I20260619 02:49:43.913002 24373 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42173
I20260619 02:49:43.913080 24373 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:43.913239 24373 heartbeater.cc:511] Master 127.21.164.254:42173 requested a full tablet report, sending...
I20260619 02:49:43.913591 23890 ts_manager.cc:194] Registered new tserver with Master: 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457)
I20260619 02:49:43.914023 23890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:35327
I20260619 02:49:43.918633 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 24242
I20260619 02:49:43.918730 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:43.918864 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.913961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:56335 (local address 127.21.164.254:42173)
0619 02:49:43.914013 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.914015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.914136 (+ 121us) server_negotiation.cc:407] Connection header received
0619 02:49:43.914212 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.914213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.914222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.914263 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.914577 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.915045 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.915707 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.915906 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.915959 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.916800 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.916802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.916802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.916808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.917903 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.918181 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.918182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.918183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.918231 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.918408 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.918409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.918409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.918512 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.918565 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.918694 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.918764 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:43.918872 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.913910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42173 (local address 127.21.164.195:56335)
0619 02:49:43.914076 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.914085 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.914158 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.914266 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.914267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.914452 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:43.914509 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.914511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.915128 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.915129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.915640 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.915642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.915961 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.915962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.916031 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.916429 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:43.916433 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:43.916723 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:43.917971 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.917972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.917973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.918125 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.918281 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:43.918281 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:43.918282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:43.918358 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:43.918562 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:43.918563 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:43.918607 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.918666 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.918765 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":111,"threads_started":1}
I20260619 02:49:43.918921 22163 external_mini_cluster.cc:1428] Setting key 0f48d21379cb5e61b8512b830004ccc5
I20260619 02:49:43.919749 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:49:43 dist-test-slave-jt0t krb5kdc[23845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:49:43 dist-test-slave-jt0t krb5kdc[23845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:43.941119 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.934559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.21.164.254:42173)
0619 02:49:43.934615 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.934617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.934626 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:43.934689 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.934690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.934700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.934748 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:43.935316 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.935798 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.936527 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.936769 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.936827 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.938920 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:43.938922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:43.938922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:43.938929 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:43.940101 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.940418 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.940419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.940419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.940471 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:43.940625 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:43.940625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:43.940626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:43.940760 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:43.940854 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.940951 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.941026 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:43.942699 23890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:43.944325 23890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:43.950579 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.946708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.21.164.195:33457)
0619 02:49:43.946937 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.946942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.946956 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:49:43.946997 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.946999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.947015 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.947121 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:43.947337 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.948459 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.949684 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.950188 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.950252 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.950309 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.950438 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.950479 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":66,"threads_started":1}
I20260619 02:49:43.950573 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.946562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:44941 (local address 127.0.0.1:34276)
0619 02:49:43.946849 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.946857 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.946894 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.947390 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.947391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.947393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:43.947439 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.947440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.948257 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.948257 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.949553 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.949556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.950246 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.950247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.950324 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.950337 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.950403 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.950460 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":5,"thread_start_us":62,"threads_started":1}
I20260619 02:49:43.950872 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.946657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.21.164.193:44941)
0619 02:49:43.947246 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.947250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.947259 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:43.947294 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.947296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.947310 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.947377 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:43.947499 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.948185 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.949638 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.950167 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.950246 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.950309 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.950735 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.950785 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":96,"threads_started":1}
I20260619 02:49:43.950575 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.946562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:33457 (local address 127.0.0.1:43720)
0619 02:49:43.946756 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.946802 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.946867 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.947124 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.947127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.947130 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:43.947206 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.947208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.948543 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.948544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.949615 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.949617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.950260 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.950261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.950345 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.950348 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.950417 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.950462 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":64,"threads_started":1}
I20260619 02:49:43.951305 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.946761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:45587 (local address 127.0.0.1:53200)
0619 02:49:43.946961 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:43.946967 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:43.946997 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:43.947265 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:43.947266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:43.947268 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:43.947307 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.947309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.948723 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.948723 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.949601 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:43.949604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.951040 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.951041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:43.951110 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.951114 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:43.951162 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:49:43.951214 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":47,"threads_started":1}
I20260619 02:49:43.951529 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:43.946890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.21.164.194:45587)
0619 02:49:43.947066 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:43.947071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:43.947079 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:43.947111 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:43.947113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:43.947128 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:43.947198 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:43.947375 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.948459 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.950344 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:43.950969 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:43.951040 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:43.951084 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:43.951403 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 02:49:43.951443 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260619 02:49:43.951674 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2d364359044db9968fdb5b47bb6d44 (DEFAULT_TABLE table=test-table [id=fc19673c51d445cdb15a048bc7358baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:43.951735 24307 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2d364359044db9968fdb5b47bb6d44 (DEFAULT_TABLE table=test-table [id=fc19673c51d445cdb15a048bc7358baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:43.952015 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2d364359044db9968fdb5b47bb6d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:43.952046 24307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2d364359044db9968fdb5b47bb6d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:43.953392 24167 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2d364359044db9968fdb5b47bb6d44 (DEFAULT_TABLE table=test-table [id=fc19673c51d445cdb15a048bc7358baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:43.953764 24167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2d364359044db9968fdb5b47bb6d44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:43.954756 24393 tablet_bootstrap.cc:492] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9: Bootstrap starting.
I20260619 02:49:43.955521 24394 tablet_bootstrap.cc:492] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e: Bootstrap starting.
I20260619 02:49:43.955852 24393 tablet_bootstrap.cc:654] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:43.956131 24394 tablet_bootstrap.cc:654] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:43.956254 24393 log.cc:824] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:43.956442 24394 log.cc:824] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:43.957093 24393 tablet_bootstrap.cc:492] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9: No bootstrap required, opened a new log
I20260619 02:49:43.957147 24393 ts_tablet_manager.cc:1397] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:43.957265 24394 tablet_bootstrap.cc:492] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e: No bootstrap required, opened a new log
I20260619 02:49:43.957389 24394 ts_tablet_manager.cc:1397] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:43.957433 24397 tablet_bootstrap.cc:492] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: Bootstrap starting.
I20260619 02:49:43.958041 24397 tablet_bootstrap.cc:654] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:43.958345 24397 log.cc:824] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:43.959118 24397 tablet_bootstrap.cc:492] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: No bootstrap required, opened a new log
I20260619 02:49:43.959180 24397 ts_tablet_manager.cc:1397] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:43.959115 24393 raft_consensus.cc:348] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } }
I20260619 02:49:43.959307 24393 raft_consensus.cc:374] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:43.959339 24393 raft_consensus.cc:729] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c580b8d4554aa38b61570f591243e9, State: Initialized, Role: FOLLOWER
I20260619 02:49:43.959019 24394 raft_consensus.cc:348] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } }
I20260619 02:49:43.959461 24393 consensus_queue.cc:260] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } }
I20260619 02:49:43.959507 24394 raft_consensus.cc:374] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:43.959553 24394 raft_consensus.cc:729] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4a5220ff004058a1e0d067c4db4b7e, State: Initialized, Role: FOLLOWER
I20260619 02:49:43.959663 24394 consensus_queue.cc:260] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } }
I20260619 02:49:43.959707 24393 ts_tablet_manager.cc:1428] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:43.959813 24373 heartbeater.cc:503] Master 127.21.164.254:42173 was elected leader, sending a full tablet report...
I20260619 02:49:43.959918 24394 ts_tablet_manager.cc:1428] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:43.959991 24093 heartbeater.cc:503] Master 127.21.164.254:42173 was elected leader, sending a full tablet report...
I20260619 02:49:43.961406 24397 raft_consensus.cc:348] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } }
I20260619 02:49:43.961534 24397 raft_consensus.cc:374] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:43.961555 24397 raft_consensus.cc:729] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a99e4eb38b443e8ec6770ab3425739, State: Initialized, Role: FOLLOWER
I20260619 02:49:43.961638 24397 consensus_queue.cc:260] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } }
I20260619 02:49:43.961864 24397 ts_tablet_manager.cc:1428] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 02:49:43.961905 24233 heartbeater.cc:503] Master 127.21.164.254:42173 was elected leader, sending a full tablet report...
W20260619 02:49:44.006191 24234 tablet.cc:2369] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:44.043565 24400 raft_consensus.cc:482] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:44.043713 24400 raft_consensus.cc:504] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } }
I20260619 02:49:44.044052 24400 leader_election.cc:288] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457), 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587)
I20260619 02:49:44.047799 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.044283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:38823 (local address 127.21.164.195:33457)
0619 02:49:44.044335 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.044337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.044348 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:44.044383 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.044384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.044393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.044457 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:44.044629 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.045434 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.046883 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.047450 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.047516 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.047522 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.047687 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.047723 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:49:44.047799 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.044445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:34461 (local address 127.21.164.194:45587)
0619 02:49:44.044490 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.044492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.044614 (+ 122us) server_negotiation.cc:407] Connection header received
0619 02:49:44.044676 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.044677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.044686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.044733 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:44.044870 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.045521 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.046883 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.047435 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.047513 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.047522 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.047687 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.047724 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:49:44.047824 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.044218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:33457 (local address 127.21.164.193:38823)
0619 02:49:44.044277 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.044289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.044335 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.044472 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.044473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.044475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:44.044543 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.044545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.045539 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.045540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.046777 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.046780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.047519 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.047520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.047623 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.047625 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.047675 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.047721 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":5}
I20260619 02:49:44.047837 24402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.044325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:45587 (local address 127.21.164.193:34461)
0619 02:49:44.044549 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.044559 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.044607 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.044742 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.044743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.044744 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:44.044799 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.044801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.045600 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.045601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.046777 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.046780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.047519 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.047520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.047614 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.047618 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.047674 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.047721 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":58,"thread_start_us":135,"threads_started":1}
I20260619 02:49:44.048243 24187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2d364359044db9968fdb5b47bb6d44" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a99e4eb38b443e8ec6770ab3425739" is_pre_election: true
I20260619 02:49:44.048271 24327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2d364359044db9968fdb5b47bb6d44" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c580b8d4554aa38b61570f591243e9" is_pre_election: true
I20260619 02:49:44.048422 24187 raft_consensus.cc:2454] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 0.
I20260619 02:49:44.048422 24327 raft_consensus.cc:2454] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 0.
I20260619 02:49:44.048676 23981 leader_election.cc:302] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a5220ff004058a1e0d067c4db4b7e, 75a99e4eb38b443e8ec6770ab3425739; no voters:
I20260619 02:49:44.048817 24400 raft_consensus.cc:2781] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:44.048890 24400 raft_consensus.cc:482] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:44.048907 24400 raft_consensus.cc:3037] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.049594 24400 raft_consensus.cc:504] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } }
I20260619 02:49:44.049710 24400 leader_election.cc:288] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 election: Requested vote from peers 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457), 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587)
I20260619 02:49:44.049904 24327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2d364359044db9968fdb5b47bb6d44" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c580b8d4554aa38b61570f591243e9"
I20260619 02:49:44.049944 24187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2d364359044db9968fdb5b47bb6d44" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a99e4eb38b443e8ec6770ab3425739"
I20260619 02:49:44.049981 24327 raft_consensus.cc:3037] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.050036 24187 raft_consensus.cc:3037] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.050657 24327 raft_consensus.cc:2454] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 1.
I20260619 02:49:44.050783 24187 raft_consensus.cc:2454] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 1.
I20260619 02:49:44.050848 23981 leader_election.cc:302] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a5220ff004058a1e0d067c4db4b7e, 74c580b8d4554aa38b61570f591243e9; no voters:
I20260619 02:49:44.050935 24400 raft_consensus.cc:2781] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:44.051097 24400 raft_consensus.cc:686] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 LEADER]: Becoming Leader. State: Replica: 1d4a5220ff004058a1e0d067c4db4b7e, State: Running, Role: LEADER
I20260619 02:49:44.051192 24400 consensus_queue.cc:237] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } }
I20260619 02:49:44.052055 23889 catalog_manager.cc:5495] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193). New cstate: current_term: 1 leader_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } health_report { overall_health: HEALTHY } } }
Jun 19 02:49:44 dist-test-slave-jt0t krb5kdc[23845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837383, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:44.095782 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.089331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42173 (local address 127.0.0.1:44454)
0619 02:49:44.089498 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.089507 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.089577 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.089709 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.089709 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.090047 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:44.090100 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.090104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.090732 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.090733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.091264 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.091267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.091576 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.091577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.091622 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.092107 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:44.092111 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:44.093456 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:44.094862 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.094869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.094872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.095055 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.095213 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:44.095213 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:44.095214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:44.095267 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:44.095490 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:44.095497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:44.095543 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.095607 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.095703 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260619 02:49:44.095868 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.089405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.21.164.254:42173)
0619 02:49:44.089462 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.089464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.089568 (+ 104us) server_negotiation.cc:407] Connection header received
0619 02:49:44.089636 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.089637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.089647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.089693 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:44.090184 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.090661 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.091348 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.091519 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.091572 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.093534 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:44.093535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:44.093535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:44.093542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:44.094783 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.095114 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.095115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.095115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.095162 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:44.095311 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:44.095312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:44.095313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:44.095420 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:44.095473 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.095677 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.095763 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:49:44.097890 23889 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:44454:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:49:44.098137 23889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:44.099514 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf94d88b0f042c2bd64171796c91b2b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=13c7a152a6a74e4988a8a3cfc3f8d3cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:44.099560 24167 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf94d88b0f042c2bd64171796c91b2b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=13c7a152a6a74e4988a8a3cfc3f8d3cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:44.099584 24307 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf94d88b0f042c2bd64171796c91b2b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=13c7a152a6a74e4988a8a3cfc3f8d3cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:49:44.099685 24167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf94d88b0f042c2bd64171796c91b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:44.099684 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf94d88b0f042c2bd64171796c91b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:44.099702 24307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf94d88b0f042c2bd64171796c91b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:44.100750 24397 tablet_bootstrap.cc:492] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739: Bootstrap starting.
I20260619 02:49:44.101161 24393 tablet_bootstrap.cc:492] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: Bootstrap starting.
I20260619 02:49:44.101167 24394 tablet_bootstrap.cc:492] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: Bootstrap starting.
I20260619 02:49:44.101361 24397 tablet_bootstrap.cc:654] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:44.101608 24393 tablet_bootstrap.cc:654] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:44.101821 24394 tablet_bootstrap.cc:654] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:44.102133 24397 tablet_bootstrap.cc:492] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739: No bootstrap required, opened a new log
I20260619 02:49:44.102183 24397 ts_tablet_manager.cc:1397] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:44.102306 24393 tablet_bootstrap.cc:492] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: No bootstrap required, opened a new log
I20260619 02:49:44.102351 24393 ts_tablet_manager.cc:1397] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:44.102336 24397 raft_consensus.cc:348] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.102408 24397 raft_consensus.cc:374] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:44.102427 24397 raft_consensus.cc:729] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a99e4eb38b443e8ec6770ab3425739, State: Initialized, Role: FOLLOWER
I20260619 02:49:44.102455 24394 tablet_bootstrap.cc:492] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: No bootstrap required, opened a new log
I20260619 02:49:44.102491 24394 ts_tablet_manager.cc:1397] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:44.102474 24397 consensus_queue.cc:260] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.102471 24393 raft_consensus.cc:348] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.102540 24393 raft_consensus.cc:374] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:44.102559 24393 raft_consensus.cc:729] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c580b8d4554aa38b61570f591243e9, State: Initialized, Role: FOLLOWER
I20260619 02:49:44.102564 24397 ts_tablet_manager.cc:1428] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 02:49:44.102593 24393 consensus_queue.cc:260] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.102617 24394 raft_consensus.cc:348] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.102670 24393 ts_tablet_manager.cc:1428] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:44.102679 24394 raft_consensus.cc:374] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:44.102697 24394 raft_consensus.cc:729] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4a5220ff004058a1e0d067c4db4b7e, State: Initialized, Role: FOLLOWER
I20260619 02:49:44.102634 24401 tablet_replica.cc:439] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } }
I20260619 02:49:44.102726 24401 tablet_replica.cc:442] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:44.102731 24394 consensus_queue.cc:260] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.102825 24394 ts_tablet_manager.cc:1428] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:44.102866 24400 tablet_replica.cc:439] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } }
I20260619 02:49:44.102939 24400 tablet_replica.cc:442] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:44.102972 24399 tablet_replica.cc:439] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } }
I20260619 02:49:44.103050 24399 tablet_replica.cc:442] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 02:49:44.116758 24094 tablet.cc:2369] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:49:44.160890 24374 tablet.cc:2369] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:44.445863 24399 raft_consensus.cc:482] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:44.445964 24399 raft_consensus.cc:504] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.446322 24399 leader_election.cc:288] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193:44941), 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587)
I20260619 02:49:44.449079 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.446576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:56005 (local address 127.21.164.194:45587)
0619 02:49:44.446624 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.446626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.446722 (+ 96us) server_negotiation.cc:407] Connection header received
0619 02:49:44.446774 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.446775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.446782 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.446830 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:44.446961 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.447433 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.448402 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.448775 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.448836 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.448844 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.448975 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.449009 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:49:44.449079 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.446555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:44651 (local address 127.21.164.193:44941)
0619 02:49:44.446615 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.446617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.446697 (+ 80us) server_negotiation.cc:407] Connection header received
0619 02:49:44.446750 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.446751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.446761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.446822 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:44.446988 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.447449 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.448393 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.448767 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.448836 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.448845 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.448966 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.449004 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:49:44.449102 24414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.446494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:44941 (local address 127.21.164.195:44651)
0619 02:49:44.446617 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.446629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.446694 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.446824 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.446824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.446827 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:44.446923 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.446925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.447519 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.447520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.448307 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.448309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.448836 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.448837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.448905 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.448909 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.448953 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.449001 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":17,"thread_start_us":56,"threads_started":1}
I20260619 02:49:44.449108 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.446503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:45587 (local address 127.21.164.195:56005)
0619 02:49:44.446644 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.446654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.446722 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.446845 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.446846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.446848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:44.446903 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.446904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.447513 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.447514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.448315 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.448317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.448846 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.448847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.448911 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.448914 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.448961 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.449004 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":32,"thread_start_us":51,"threads_started":1}
I20260619 02:49:44.449383 24187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf94d88b0f042c2bd64171796c91b2b" candidate_uuid: "74c580b8d4554aa38b61570f591243e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a99e4eb38b443e8ec6770ab3425739" is_pre_election: true
I20260619 02:49:44.449417 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf94d88b0f042c2bd64171796c91b2b" candidate_uuid: "74c580b8d4554aa38b61570f591243e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" is_pre_election: true
I20260619 02:49:44.449470 24187 raft_consensus.cc:2454] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c580b8d4554aa38b61570f591243e9 in term 0.
I20260619 02:49:44.449515 24047 raft_consensus.cc:2454] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c580b8d4554aa38b61570f591243e9 in term 0.
I20260619 02:49:44.449648 24261 leader_election.cc:302] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c580b8d4554aa38b61570f591243e9, 75a99e4eb38b443e8ec6770ab3425739; no voters:
I20260619 02:49:44.449779 24399 raft_consensus.cc:2781] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:44.449819 24399 raft_consensus.cc:482] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:44.449838 24399 raft_consensus.cc:3037] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.450508 24399 raft_consensus.cc:504] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.450631 24399 leader_election.cc:288] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [CANDIDATE]: Term 1 election: Requested vote from peers 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193:44941), 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587)
I20260619 02:49:44.450765 24187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf94d88b0f042c2bd64171796c91b2b" candidate_uuid: "74c580b8d4554aa38b61570f591243e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a99e4eb38b443e8ec6770ab3425739"
I20260619 02:49:44.450839 24187 raft_consensus.cc:3037] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.450855 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf94d88b0f042c2bd64171796c91b2b" candidate_uuid: "74c580b8d4554aa38b61570f591243e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4a5220ff004058a1e0d067c4db4b7e"
I20260619 02:49:44.450956 24047 raft_consensus.cc:3037] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.451522 24187 raft_consensus.cc:2454] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c580b8d4554aa38b61570f591243e9 in term 1.
I20260619 02:49:44.451647 24047 raft_consensus.cc:2454] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c580b8d4554aa38b61570f591243e9 in term 1.
I20260619 02:49:44.451658 24261 leader_election.cc:302] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c580b8d4554aa38b61570f591243e9, 75a99e4eb38b443e8ec6770ab3425739; no voters:
I20260619 02:49:44.451728 24399 raft_consensus.cc:2781] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:44.451833 24399 raft_consensus.cc:686] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [term 1 LEADER]: Becoming Leader. State: Replica: 74c580b8d4554aa38b61570f591243e9, State: Running, Role: LEADER
I20260619 02:49:44.451920 24399 consensus_queue.cc:237] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.452306 24416 tablet_replica.cc:439] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: TxnStatusTablet state changed. Reason: New leader 74c580b8d4554aa38b61570f591243e9. Latest consensus state: current_term: 1 leader_uuid: "74c580b8d4554aa38b61570f591243e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } }
I20260619 02:49:44.452371 24416 tablet_replica.cc:442] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:44.452526 24418 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:49:44.452642 24418 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:49:44.452632 23890 catalog_manager.cc:5495] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c580b8d4554aa38b61570f591243e9 (127.21.164.195). New cstate: current_term: 1 leader_uuid: "74c580b8d4554aa38b61570f591243e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } health_report { overall_health: HEALTHY } } }
I20260619 02:49:44.457473 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.455234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.21.164.195:33457)
0619 02:49:44.455289 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.455291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.455298 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:49:44.455326 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.455327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.455344 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.455396 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:44.455493 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.455954 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.456732 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.456959 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.457012 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.457173 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:44.457266 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:44.457331 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.457384 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.457405 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 02:49:44.457491 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.455153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:33457 (local address 127.0.0.1:43728)
0619 02:49:44.455212 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.455223 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.455266 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.455398 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.455399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.455400 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:44.455447 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.455450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.456022 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.456023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.456642 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.456670 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.457016 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.457017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.457084 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.457101 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:44.457317 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:44.457324 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.457378 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.457427 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 02:49:44.459177 24187 raft_consensus.cc:1261] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Refusing update from remote peer 74c580b8d4554aa38b61570f591243e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:44.459307 24047 raft_consensus.cc:1261] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Refusing update from remote peer 74c580b8d4554aa38b61570f591243e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:44.459578 24399 consensus_queue.cc:1035] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:44.459687 24416 consensus_queue.cc:1035] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:44.460788 24400 tablet_replica.cc:439] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: TxnStatusTablet state changed. Reason: New leader 74c580b8d4554aa38b61570f591243e9. Latest consensus state: current_term: 1 leader_uuid: "74c580b8d4554aa38b61570f591243e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } }
I20260619 02:49:44.460887 24400 tablet_replica.cc:442] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:44.460961 24401 tablet_replica.cc:439] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739: TxnStatusTablet state changed. Reason: New leader 74c580b8d4554aa38b61570f591243e9. Latest consensus state: current_term: 1 leader_uuid: "74c580b8d4554aa38b61570f591243e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } }
I20260619 02:49:44.461036 24401 tablet_replica.cc:442] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:44.461998 24416 tablet_replica.cc:439] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74c580b8d4554aa38b61570f591243e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } }
I20260619 02:49:44.462073 24416 tablet_replica.cc:442] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:44.462045 24399 tablet_replica.cc:439] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74c580b8d4554aa38b61570f591243e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } }
I20260619 02:49:44.462124 24399 tablet_replica.cc:442] T 6cf94d88b0f042c2bd64171796c91b2b P 74c580b8d4554aa38b61570f591243e9: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:49:44.462239 24421 mvcc.cc:205] Tried to move back new op lower bound from 7298405926742405120 to 7298405926715592704. Current Snapshot: MvccSnapshot[applied={T|T < 7298405926742405120}]
I20260619 02:49:44.462324 24401 tablet_replica.cc:439] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74c580b8d4554aa38b61570f591243e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } }
I20260619 02:49:44.462400 24401 tablet_replica.cc:442] T 6cf94d88b0f042c2bd64171796c91b2b P 75a99e4eb38b443e8ec6770ab3425739: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:44.462770 24403 tablet_replica.cc:439] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74c580b8d4554aa38b61570f591243e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } } }
I20260619 02:49:44.462844 24403 tablet_replica.cc:442] T 6cf94d88b0f042c2bd64171796c91b2b P 1d4a5220ff004058a1e0d067c4db4b7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:49:44.463495 24403 consensus_queue.cc:1035] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:44.465006 24403 consensus_queue.cc:1035] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:44.465261 24432 mvcc.cc:205] Tried to move back new op lower bound from 7298405926742405120 to 7298405926715592704. Current Snapshot: MvccSnapshot[applied={T|T < 7298405926742405120}]
I20260619 02:49:44.471174 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.464333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.21.164.193:44941)
0619 02:49:44.464375 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.464377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.464387 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:44.464434 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.464435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.464451 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.464511 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:44.468872 (+ 4361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.469636 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.470366 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.470672 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.470734 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.470852 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:44.470981 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:44.471058 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.471074 (+ 16us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.471097 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:49:44.476151 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.473576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:38293 (local address 127.21.164.195:33457)
0619 02:49:44.473616 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.473617 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.473661 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:49:44.473732 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.473733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.473742 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.473782 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:44.473901 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.474336 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.475444 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.475628 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.475680 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.475906 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:44.475947 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:44.476013 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.476074 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.476093 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:49:44.476214 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.473481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:33457 (local address 127.21.164.193:38293)
0619 02:49:44.473604 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.473616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.473677 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.473786 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.473787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.473788 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:44.473848 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.473849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.474413 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.474413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.475356 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.475358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.475693 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.475694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.475761 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.475777 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:44.476023 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:44.476024 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.476066 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.476114 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260619 02:49:44.485801 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.483162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:55477 (local address 127.21.164.193:44941)
0619 02:49:44.483208 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:44.483211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:44.483344 (+ 133us) server_negotiation.cc:407] Connection header received
0619 02:49:44.483400 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:44.483400 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:44.483408 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:44.483448 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:44.483559 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.484016 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.485196 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.485401 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.485451 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.485616 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:44.485642 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:44.485671 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:44.485726 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 02:49:44.485743 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:49:44.485913 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:44.483128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:44941 (local address 127.21.164.195:55477)
0619 02:49:44.483274 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:44.483285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:44.483354 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:44.483452 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:44.483453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:44.483454 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:44.483516 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.483517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.484077 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.484078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.485128 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:44.485130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:44.485466 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:44.485467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:44.485534 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:44.485550 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:44.485677 (+ 127us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:44.485679 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:49:44.485723 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:49:44.485764 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260619 02:49:44.494971 23889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:44.495091 23889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:44.495877 23889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f: Sending DeleteTablet for 3 replicas of tablet 8e2d364359044db9968fdb5b47bb6d44
I20260619 02:49:44.496166 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2d364359044db9968fdb5b47bb6d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:44 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:34276
I20260619 02:49:44.496346 24167 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2d364359044db9968fdb5b47bb6d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:44 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:53200
I20260619 02:49:44.496583 24444 tablet_replica.cc:330] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: stopping tablet replica
I20260619 02:49:44.496722 24307 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2d364359044db9968fdb5b47bb6d44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:44 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:43720
I20260619 02:49:44.496802 24444 raft_consensus.cc:2229] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:44.496873 24444 raft_consensus.cc:2258] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:44.497088 23889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:44.497223 23889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:44.497306 24444 ts_tablet_manager.cc:1905] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:44.498454 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 953277244cd1461f9b2ff50520da0685 (DEFAULT_TABLE table=test-table [id=e734dbdff4f147319ae2092e54364048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:44.498574 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953277244cd1461f9b2ff50520da0685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:44.499059 24444 ts_tablet_manager.cc:1918] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:44.499122 24444 log.cc:1196] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/8e2d364359044db9968fdb5b47bb6d44
I20260619 02:49:44.499361 24444 ts_tablet_manager.cc:1939] T 8e2d364359044db9968fdb5b47bb6d44 P 75a99e4eb38b443e8ec6770ab3425739: Deleting consensus metadata
I20260619 02:49:44.499845 24394 tablet_bootstrap.cc:492] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e: Bootstrap starting.
I20260619 02:49:44.500226 24394 tablet_bootstrap.cc:654] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:44.500891 24394 tablet_bootstrap.cc:492] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e: No bootstrap required, opened a new log
I20260619 02:49:44.500941 24394 ts_tablet_manager.cc:1397] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:44.501026 23877 catalog_manager.cc:4841] TS 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587): tablet 8e2d364359044db9968fdb5b47bb6d44 (table test-table [id=fc19673c51d445cdb15a048bc7358baa]) successfully deleted
I20260619 02:49:44.501070 24394 raft_consensus.cc:348] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.501132 24394 raft_consensus.cc:374] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:44.501150 24394 raft_consensus.cc:729] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4a5220ff004058a1e0d067c4db4b7e, State: Initialized, Role: FOLLOWER
I20260619 02:49:44.501190 24394 consensus_queue.cc:260] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.501279 24394 ts_tablet_manager.cc:1428] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:44.501585 24167 tablet_service.cc:1467] Processing CreateTablet for tablet 953277244cd1461f9b2ff50520da0685 (DEFAULT_TABLE table=test-table [id=e734dbdff4f147319ae2092e54364048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:44.501691 24167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953277244cd1461f9b2ff50520da0685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:44.502887 24397 tablet_bootstrap.cc:492] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739: Bootstrap starting.
I20260619 02:49:44.503261 24397 tablet_bootstrap.cc:654] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:44.503711 24307 tablet_service.cc:1467] Processing CreateTablet for tablet 953277244cd1461f9b2ff50520da0685 (DEFAULT_TABLE table=test-table [id=e734dbdff4f147319ae2092e54364048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:44.503823 24307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953277244cd1461f9b2ff50520da0685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:44.503890 24397 tablet_bootstrap.cc:492] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739: No bootstrap required, opened a new log
I20260619 02:49:44.503931 24397 ts_tablet_manager.cc:1397] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:44.504058 24397 raft_consensus.cc:348] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.504117 24397 raft_consensus.cc:374] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:44.504136 24397 raft_consensus.cc:729] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a99e4eb38b443e8ec6770ab3425739, State: Initialized, Role: FOLLOWER
I20260619 02:49:44.504179 24397 consensus_queue.cc:260] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.504251 24397 ts_tablet_manager.cc:1428] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:44.504752 24443 tablet_replica.cc:330] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e: stopping tablet replica
I20260619 02:49:44.504881 24443 raft_consensus.cc:2229] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:44.504988 24443 raft_consensus.cc:2258] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:44.505259 24393 tablet_bootstrap.cc:492] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9: Bootstrap starting.
I20260619 02:49:44.505431 24443 ts_tablet_manager.cc:1905] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:44.505738 24393 tablet_bootstrap.cc:654] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:44.508272 24393 tablet_bootstrap.cc:492] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9: No bootstrap required, opened a new log
I20260619 02:49:44.508356 24393 ts_tablet_manager.cc:1397] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 02:49:44.509061 24393 raft_consensus.cc:348] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.509182 24393 raft_consensus.cc:374] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:44.509205 24393 raft_consensus.cc:729] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c580b8d4554aa38b61570f591243e9, State: Initialized, Role: FOLLOWER
I20260619 02:49:44.509383 24393 consensus_queue.cc:260] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.510156 24393 ts_tablet_manager.cc:1428] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:44.511227 24445 tablet_replica.cc:330] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9: stopping tablet replica
I20260619 02:49:44.511439 24445 raft_consensus.cc:2229] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:44.511590 24445 raft_consensus.cc:2258] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:44.512156 24445 ts_tablet_manager.cc:1905] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:44.512573 24443 ts_tablet_manager.cc:1918] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:44.512691 24443 log.cc:1196] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/wal/wals/8e2d364359044db9968fdb5b47bb6d44
I20260619 02:49:44.512979 24443 ts_tablet_manager.cc:1939] T 8e2d364359044db9968fdb5b47bb6d44 P 1d4a5220ff004058a1e0d067c4db4b7e: Deleting consensus metadata
I20260619 02:49:44.513561 23875 catalog_manager.cc:4841] TS 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193:44941): tablet 8e2d364359044db9968fdb5b47bb6d44 (table test-table [id=fc19673c51d445cdb15a048bc7358baa]) successfully deleted
I20260619 02:49:44.514957 24445 ts_tablet_manager.cc:1918] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:44.515076 24445 log.cc:1196] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/8e2d364359044db9968fdb5b47bb6d44
I20260619 02:49:44.515378 24445 ts_tablet_manager.cc:1939] T 8e2d364359044db9968fdb5b47bb6d44 P 74c580b8d4554aa38b61570f591243e9: Deleting consensus metadata
I20260619 02:49:44.515799 23877 catalog_manager.cc:4841] TS 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457): tablet 8e2d364359044db9968fdb5b47bb6d44 (table test-table [id=fc19673c51d445cdb15a048bc7358baa]) successfully deleted
I20260619 02:49:44.674911 24403 raft_consensus.cc:482] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:44.675009 24403 raft_consensus.cc:504] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.675148 24403 leader_election.cc:288] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587), 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457)
I20260619 02:49:44.675379 24187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953277244cd1461f9b2ff50520da0685" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a99e4eb38b443e8ec6770ab3425739" is_pre_election: true
I20260619 02:49:44.675421 24327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953277244cd1461f9b2ff50520da0685" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c580b8d4554aa38b61570f591243e9" is_pre_election: true
I20260619 02:49:44.675493 24327 raft_consensus.cc:2454] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 0.
I20260619 02:49:44.675493 24187 raft_consensus.cc:2454] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 0.
I20260619 02:49:44.675642 23981 leader_election.cc:302] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a5220ff004058a1e0d067c4db4b7e, 74c580b8d4554aa38b61570f591243e9; no voters:
I20260619 02:49:44.675751 24403 raft_consensus.cc:2781] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:44.675803 24403 raft_consensus.cc:482] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:44.675822 24403 raft_consensus.cc:3037] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.676540 24403 raft_consensus.cc:504] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.676674 24403 leader_election.cc:288] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 election: Requested vote from peers 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587), 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457)
I20260619 02:49:44.676811 24187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953277244cd1461f9b2ff50520da0685" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a99e4eb38b443e8ec6770ab3425739"
I20260619 02:49:44.676883 24187 raft_consensus.cc:3037] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.676868 24327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953277244cd1461f9b2ff50520da0685" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c580b8d4554aa38b61570f591243e9"
I20260619 02:49:44.676950 24327 raft_consensus.cc:3037] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:44.677470 24187 raft_consensus.cc:2454] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 1.
I20260619 02:49:44.677546 24327 raft_consensus.cc:2454] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 1.
I20260619 02:49:44.677623 23981 leader_election.cc:302] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a5220ff004058a1e0d067c4db4b7e, 75a99e4eb38b443e8ec6770ab3425739; no voters:
I20260619 02:49:44.677711 24403 raft_consensus.cc:2781] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:44.677770 24403 raft_consensus.cc:686] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 LEADER]: Becoming Leader. State: Replica: 1d4a5220ff004058a1e0d067c4db4b7e, State: Running, Role: LEADER
I20260619 02:49:44.677820 24403 consensus_queue.cc:237] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:44.678364 23889 catalog_manager.cc:5495] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193). New cstate: current_term: 1 leader_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:44.692871 24327 raft_consensus.cc:1261] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Refusing update from remote peer 1d4a5220ff004058a1e0d067c4db4b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:44.692883 24187 raft_consensus.cc:1261] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Refusing update from remote peer 1d4a5220ff004058a1e0d067c4db4b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:44.693101 24403 consensus_queue.cc:1035] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:44.693200 24400 consensus_queue.cc:1035] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:49:44.698139 24422 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:49:46.216553 23890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:46.216670 23890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:46.217383 23890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f: Sending DeleteTablet for 3 replicas of tablet 953277244cd1461f9b2ff50520da0685
I20260619 02:49:46.217628 24307 tablet_service.cc:1514] Processing DeleteTablet for tablet 953277244cd1461f9b2ff50520da0685 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:46 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:43720
I20260619 02:49:46.217633 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 953277244cd1461f9b2ff50520da0685 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:46 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:34276
I20260619 02:49:46.217799 24167 tablet_service.cc:1514] Processing DeleteTablet for tablet 953277244cd1461f9b2ff50520da0685 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:46 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:53200
I20260619 02:49:46.217818 24474 tablet_replica.cc:330] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e: stopping tablet replica
I20260619 02:49:46.217867 24474 raft_consensus.cc:2229] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:46.217916 24474 raft_consensus.cc:2258] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:46.218113 24474 ts_tablet_manager.cc:1905] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:46.218222 24473 tablet_replica.cc:330] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9: stopping tablet replica
I20260619 02:49:46.218297 24473 raft_consensus.cc:2229] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:46.218353 24473 raft_consensus.cc:2258] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:46.218567 24473 ts_tablet_manager.cc:1905] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:46.219815 24474 ts_tablet_manager.cc:1918] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:49:46.219878 24474 log.cc:1196] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/wal/wals/953277244cd1461f9b2ff50520da0685
I20260619 02:49:46.220104 24474 ts_tablet_manager.cc:1939] T 953277244cd1461f9b2ff50520da0685 P 1d4a5220ff004058a1e0d067c4db4b7e: Deleting consensus metadata
I20260619 02:49:46.220158 24473 ts_tablet_manager.cc:1918] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:49:46.220232 24473 log.cc:1196] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/953277244cd1461f9b2ff50520da0685
I20260619 02:49:46.220427 24473 ts_tablet_manager.cc:1939] T 953277244cd1461f9b2ff50520da0685 P 74c580b8d4554aa38b61570f591243e9: Deleting consensus metadata
I20260619 02:49:46.220726 24475 tablet_replica.cc:330] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739: stopping tablet replica
I20260619 02:49:46.220752 23877 catalog_manager.cc:4841] TS 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457): tablet 953277244cd1461f9b2ff50520da0685 (table test-table [id=e734dbdff4f147319ae2092e54364048]) successfully deleted
I20260619 02:49:46.220795 24475 raft_consensus.cc:2229] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:46.220830 24475 raft_consensus.cc:2258] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:46.221050 24475 ts_tablet_manager.cc:1905] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:46.221575 23875 catalog_manager.cc:4841] TS 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193:44941): tablet 953277244cd1461f9b2ff50520da0685 (table test-table [id=e734dbdff4f147319ae2092e54364048]) successfully deleted
I20260619 02:49:46.222317 24475 ts_tablet_manager.cc:1918] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:49:46.222369 24475 log.cc:1196] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/953277244cd1461f9b2ff50520da0685
I20260619 02:49:46.222539 24475 ts_tablet_manager.cc:1939] T 953277244cd1461f9b2ff50520da0685 P 75a99e4eb38b443e8ec6770ab3425739: Deleting consensus metadata
I20260619 02:49:46.222915 23877 catalog_manager.cc:4841] TS 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587): tablet 953277244cd1461f9b2ff50520da0685 (table test-table [id=e734dbdff4f147319ae2092e54364048]) successfully deleted
Jun 19 02:49:46 dist-test-slave-jt0t krb5kdc[23845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:49:46 dist-test-slave-jt0t krb5kdc[23845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:46.241815 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.235087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.21.164.254:42173)
0619 02:49:46.235215 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:46.235217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:46.235226 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:46.235283 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:46.235284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:46.235294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:46.235351 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:46.235871 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.236382 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.237163 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.237365 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.237417 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.239514 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:46.239515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:46.239515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:46.239523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:46.240896 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:46.241196 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:46.241197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:46.241197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:46.241261 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:46.241409 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:46.241410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:46.241410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:46.241517 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:46.241590 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:46.241662 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 02:49:46.241728 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260619 02:49:46.243351 23890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:46.243500 23890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:46.244788 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbaef94b6b54cbab5a1aa59c28f14e5 (DEFAULT_TABLE table=test-table [id=5ac6d307a58740cfaa7132833a7fb066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:46.244867 24167 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbaef94b6b54cbab5a1aa59c28f14e5 (DEFAULT_TABLE table=test-table [id=5ac6d307a58740cfaa7132833a7fb066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:46.244928 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbaef94b6b54cbab5a1aa59c28f14e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:46.244921 24307 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbaef94b6b54cbab5a1aa59c28f14e5 (DEFAULT_TABLE table=test-table [id=5ac6d307a58740cfaa7132833a7fb066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:46.244964 24167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbaef94b6b54cbab5a1aa59c28f14e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:46.245041 24307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbaef94b6b54cbab5a1aa59c28f14e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:46.246083 24484 tablet_bootstrap.cc:492] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9: Bootstrap starting.
I20260619 02:49:46.246218 24485 tablet_bootstrap.cc:492] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739: Bootstrap starting.
I20260619 02:49:46.246456 24484 tablet_bootstrap.cc:654] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:46.246590 24485 tablet_bootstrap.cc:654] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:46.246646 24486 tablet_bootstrap.cc:492] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e: Bootstrap starting.
I20260619 02:49:46.246996 24486 tablet_bootstrap.cc:654] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:46.247072 24484 tablet_bootstrap.cc:492] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9: No bootstrap required, opened a new log
I20260619 02:49:46.247097 24485 tablet_bootstrap.cc:492] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739: No bootstrap required, opened a new log
I20260619 02:49:46.247117 24484 ts_tablet_manager.cc:1397] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:46.247144 24485 ts_tablet_manager.cc:1397] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:46.247259 24485 raft_consensus.cc:348] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.247244 24484 raft_consensus.cc:348] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.247326 24485 raft_consensus.cc:374] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:46.247327 24484 raft_consensus.cc:374] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:46.247339 24485 raft_consensus.cc:729] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a99e4eb38b443e8ec6770ab3425739, State: Initialized, Role: FOLLOWER
I20260619 02:49:46.247349 24484 raft_consensus.cc:729] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c580b8d4554aa38b61570f591243e9, State: Initialized, Role: FOLLOWER
I20260619 02:49:46.247390 24485 consensus_queue.cc:260] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.247401 24484 consensus_queue.cc:260] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.247490 24484 ts_tablet_manager.cc:1428] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:46.247540 24485 ts_tablet_manager.cc:1428] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:46.247782 24486 tablet_bootstrap.cc:492] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e: No bootstrap required, opened a new log
I20260619 02:49:46.247830 24486 ts_tablet_manager.cc:1397] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:46.247922 24486 raft_consensus.cc:348] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.247991 24486 raft_consensus.cc:374] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:46.248014 24486 raft_consensus.cc:729] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4a5220ff004058a1e0d067c4db4b7e, State: Initialized, Role: FOLLOWER
I20260619 02:49:46.248061 24486 consensus_queue.cc:260] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.248150 24486 ts_tablet_manager.cc:1428] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:46.320519 24467 raft_consensus.cc:482] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:46.320641 24467 raft_consensus.cc:504] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.320823 24467 leader_election.cc:288] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587), 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457)
I20260619 02:49:46.321041 24187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbaef94b6b54cbab5a1aa59c28f14e5" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a99e4eb38b443e8ec6770ab3425739" is_pre_election: true
I20260619 02:49:46.321041 24327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbaef94b6b54cbab5a1aa59c28f14e5" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c580b8d4554aa38b61570f591243e9" is_pre_election: true
I20260619 02:49:46.321135 24327 raft_consensus.cc:2454] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 0.
I20260619 02:49:46.321135 24187 raft_consensus.cc:2454] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 0.
I20260619 02:49:46.321282 23981 leader_election.cc:302] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a5220ff004058a1e0d067c4db4b7e, 74c580b8d4554aa38b61570f591243e9; no voters:
I20260619 02:49:46.321373 24467 raft_consensus.cc:2781] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:46.321413 24467 raft_consensus.cc:482] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:46.321435 24467 raft_consensus.cc:3037] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:46.322321 24467 raft_consensus.cc:504] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.322456 24467 leader_election.cc:288] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 election: Requested vote from peers 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587), 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457)
I20260619 02:49:46.322631 24327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbaef94b6b54cbab5a1aa59c28f14e5" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c580b8d4554aa38b61570f591243e9"
I20260619 02:49:46.322629 24187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbaef94b6b54cbab5a1aa59c28f14e5" candidate_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a99e4eb38b443e8ec6770ab3425739"
I20260619 02:49:46.322701 24327 raft_consensus.cc:3037] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:46.322721 24187 raft_consensus.cc:3037] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:46.323290 24327 raft_consensus.cc:2454] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 1.
I20260619 02:49:46.323299 24187 raft_consensus.cc:2454] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d4a5220ff004058a1e0d067c4db4b7e in term 1.
I20260619 02:49:46.323463 23981 leader_election.cc:302] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a5220ff004058a1e0d067c4db4b7e, 75a99e4eb38b443e8ec6770ab3425739; no voters:
I20260619 02:49:46.323565 24467 raft_consensus.cc:2781] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:46.323628 24467 raft_consensus.cc:686] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 LEADER]: Becoming Leader. State: Replica: 1d4a5220ff004058a1e0d067c4db4b7e, State: Running, Role: LEADER
I20260619 02:49:46.323678 24467 consensus_queue.cc:237] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.324268 23890 catalog_manager.cc:5495] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193). New cstate: current_term: 1 leader_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:46.343329 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.341152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.21.164.193:44941)
0619 02:49:46.341283 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:46.341285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:46.341293 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:46.341314 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:46.341315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:46.341324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:46.341379 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:46.341479 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.341958 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.342692 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.342932 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.342986 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.343076 (+ 90us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:46.343115 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:46.343159 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:46.343243 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 02:49:46.343261 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260619 02:49:46.345290 24327 raft_consensus.cc:1261] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Refusing update from remote peer 1d4a5220ff004058a1e0d067c4db4b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:46.345299 24187 raft_consensus.cc:1261] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Refusing update from remote peer 1d4a5220ff004058a1e0d067c4db4b7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:46.345501 24464 consensus_queue.cc:1035] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:46.345620 24467 consensus_queue.cc:1035] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:46.346508 24456 mvcc.cc:205] Tried to move back new op lower bound from 7298405934469091328 to 7298405934382157824. Current Snapshot: MvccSnapshot[applied={T|T < 7298405934469091328}]
I20260619 02:49:46.354826 23889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:46.354920 23889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:46.355466 23889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f: Sending DeleteTablet for 3 replicas of tablet 7dbaef94b6b54cbab5a1aa59c28f14e5
I20260619 02:49:46.355706 24307 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbaef94b6b54cbab5a1aa59c28f14e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:46 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:43720
I20260619 02:49:46.355692 24167 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbaef94b6b54cbab5a1aa59c28f14e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:46 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:53200
I20260619 02:49:46.355715 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbaef94b6b54cbab5a1aa59c28f14e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:46 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:34276
I20260619 02:49:46.355810 24475 tablet_replica.cc:330] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739: stopping tablet replica
I20260619 02:49:46.355820 24474 tablet_replica.cc:330] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e: stopping tablet replica
I20260619 02:49:46.355866 24475 raft_consensus.cc:2229] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:46.355870 24474 raft_consensus.cc:2229] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:46.355903 24475 raft_consensus.cc:2258] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:46.355916 24474 raft_consensus.cc:2258] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:46.355974 24473 tablet_replica.cc:330] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9: stopping tablet replica
I20260619 02:49:46.356030 24473 raft_consensus.cc:2229] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:46.356062 24473 raft_consensus.cc:2258] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:46.356021 23889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:46.356148 23889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:46.356343 24475 ts_tablet_manager.cc:1905] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:46.356371 24473 ts_tablet_manager.cc:1905] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:46.356629 24474 ts_tablet_manager.cc:1905] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:46.357801 24475 ts_tablet_manager.cc:1918] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:46.357849 24475 log.cc:1196] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/7dbaef94b6b54cbab5a1aa59c28f14e5
I20260619 02:49:46.357877 24473 ts_tablet_manager.cc:1918] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:46.357939 24473 log.cc:1196] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/7dbaef94b6b54cbab5a1aa59c28f14e5
I20260619 02:49:46.358072 24475 ts_tablet_manager.cc:1939] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 75a99e4eb38b443e8ec6770ab3425739: Deleting consensus metadata
I20260619 02:49:46.358080 24474 ts_tablet_manager.cc:1918] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:49:46.358124 24474 log.cc:1196] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-0/wal/wals/7dbaef94b6b54cbab5a1aa59c28f14e5
I20260619 02:49:46.358135 24473 ts_tablet_manager.cc:1939] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 74c580b8d4554aa38b61570f591243e9: Deleting consensus metadata
I20260619 02:49:46.358304 24474 ts_tablet_manager.cc:1939] T 7dbaef94b6b54cbab5a1aa59c28f14e5 P 1d4a5220ff004058a1e0d067c4db4b7e: Deleting consensus metadata
I20260619 02:49:46.358353 23877 catalog_manager.cc:4841] TS 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587): tablet 7dbaef94b6b54cbab5a1aa59c28f14e5 (table test-table [id=5ac6d307a58740cfaa7132833a7fb066]) successfully deleted
I20260619 02:49:46.358606 23877 catalog_manager.cc:4841] TS 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457): tablet 7dbaef94b6b54cbab5a1aa59c28f14e5 (table test-table [id=5ac6d307a58740cfaa7132833a7fb066]) successfully deleted
I20260619 02:49:46.358723 23875 catalog_manager.cc:4841] TS 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193:44941): tablet 7dbaef94b6b54cbab5a1aa59c28f14e5 (table test-table [id=5ac6d307a58740cfaa7132833a7fb066]) successfully deleted
I20260619 02:49:46.359519 24167 tablet_service.cc:1467] Processing CreateTablet for tablet 09cc6f56d72b49448472ca156cd705fe (DEFAULT_TABLE table=test-table [id=1f0f740e077645068a46b2da4fb85cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:46.359534 24307 tablet_service.cc:1467] Processing CreateTablet for tablet 09cc6f56d72b49448472ca156cd705fe (DEFAULT_TABLE table=test-table [id=1f0f740e077645068a46b2da4fb85cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:46.359534 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 09cc6f56d72b49448472ca156cd705fe (DEFAULT_TABLE table=test-table [id=1f0f740e077645068a46b2da4fb85cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:46.359622 24167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09cc6f56d72b49448472ca156cd705fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:46.359627 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09cc6f56d72b49448472ca156cd705fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:46.359625 24307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09cc6f56d72b49448472ca156cd705fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:46.360548 24485 tablet_bootstrap.cc:492] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739: Bootstrap starting.
I20260619 02:49:46.360737 24486 tablet_bootstrap.cc:492] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e: Bootstrap starting.
I20260619 02:49:46.360939 24485 tablet_bootstrap.cc:654] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:46.360999 24486 tablet_bootstrap.cc:654] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:46.361209 24484 tablet_bootstrap.cc:492] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9: Bootstrap starting.
I20260619 02:49:46.361451 24485 tablet_bootstrap.cc:492] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739: No bootstrap required, opened a new log
I20260619 02:49:46.361493 24485 ts_tablet_manager.cc:1397] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:46.361512 24484 tablet_bootstrap.cc:654] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:46.361517 24486 tablet_bootstrap.cc:492] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e: No bootstrap required, opened a new log
I20260619 02:49:46.361552 24486 ts_tablet_manager.cc:1397] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:49:46.361665 24486 raft_consensus.cc:348] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.361739 24486 raft_consensus.cc:374] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:46.361764 24486 raft_consensus.cc:729] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4a5220ff004058a1e0d067c4db4b7e, State: Initialized, Role: FOLLOWER
I20260619 02:49:46.361756 24485 raft_consensus.cc:348] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.361816 24486 consensus_queue.cc:260] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.361840 24485 raft_consensus.cc:374] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:46.361861 24485 raft_consensus.cc:729] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a99e4eb38b443e8ec6770ab3425739, State: Initialized, Role: FOLLOWER
I20260619 02:49:46.361905 24486 ts_tablet_manager.cc:1428] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:46.361897 24485 consensus_queue.cc:260] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.361982 24485 ts_tablet_manager.cc:1428] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:46.362319 24484 tablet_bootstrap.cc:492] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9: No bootstrap required, opened a new log
I20260619 02:49:46.362366 24484 ts_tablet_manager.cc:1397] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:46.362470 24484 raft_consensus.cc:348] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.362536 24484 raft_consensus.cc:374] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:46.362565 24484 raft_consensus.cc:729] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c580b8d4554aa38b61570f591243e9, State: Initialized, Role: FOLLOWER
I20260619 02:49:46.362605 24484 consensus_queue.cc:260] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.362690 24484 ts_tablet_manager.cc:1428] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:46.519680 24490 raft_consensus.cc:482] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:46.519786 24490 raft_consensus.cc:504] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.520195 24490 leader_election.cc:288] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193:44941), 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457)
I20260619 02:49:46.524264 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.520431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:56521 (local address 127.21.164.195:33457)
0619 02:49:46.520548 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:46.520550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:46.520561 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:46.520672 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:46.520673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:46.520684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:46.520769 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:46.520970 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.521729 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.523305 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.523906 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.523970 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.523979 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:46.524141 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 02:49:46.524186 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260619 02:49:46.524264 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.520458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:37975 (local address 127.21.164.193:44941)
0619 02:49:46.520519 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:46.520522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:46.520638 (+ 116us) server_negotiation.cc:407] Connection header received
0619 02:49:46.520691 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:46.520692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:46.520725 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:46.520775 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:46.520970 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.521734 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.523305 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.523885 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.523969 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.523979 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:46.524142 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 02:49:46.524186 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:49:46.524297 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.520381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:44941 (local address 127.21.164.194:37975)
0619 02:49:46.520563 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.520572 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:46.520626 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:46.520779 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:46.520780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:46.520784 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:46.520903 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.520905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.521835 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.521835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.523198 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.523202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.523973 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.523974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.524065 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.524074 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:49:46.524128 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:49:46.524178 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":89,"thread_start_us":101,"threads_started":1}
I20260619 02:49:46.524295 24500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.520358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:33457 (local address 127.21.164.194:56521)
0619 02:49:46.520493 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.520504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:46.520567 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:46.520778 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:46.520779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:46.520782 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:46.520903 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.520905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.521816 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.521817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.523198 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.523202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.523974 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.523975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.524065 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.524072 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:49:46.524128 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:49:46.524178 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":104,"thread_start_us":62,"threads_started":1}
I20260619 02:49:46.524573 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09cc6f56d72b49448472ca156cd705fe" candidate_uuid: "75a99e4eb38b443e8ec6770ab3425739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" is_pre_election: true
I20260619 02:49:46.524695 24047 raft_consensus.cc:2454] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75a99e4eb38b443e8ec6770ab3425739 in term 0.
I20260619 02:49:46.524714 24327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09cc6f56d72b49448472ca156cd705fe" candidate_uuid: "75a99e4eb38b443e8ec6770ab3425739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c580b8d4554aa38b61570f591243e9" is_pre_election: true
I20260619 02:49:46.524807 24327 raft_consensus.cc:2454] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75a99e4eb38b443e8ec6770ab3425739 in term 0.
I20260619 02:49:46.524868 24119 leader_election.cc:302] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a5220ff004058a1e0d067c4db4b7e, 75a99e4eb38b443e8ec6770ab3425739; no voters:
I20260619 02:49:46.524992 24490 raft_consensus.cc:2781] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:46.525025 24490 raft_consensus.cc:482] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:46.525043 24490 raft_consensus.cc:3037] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:46.525764 24490 raft_consensus.cc:504] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.525913 24490 leader_election.cc:288] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [CANDIDATE]: Term 1 election: Requested vote from peers 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193:44941), 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457)
I20260619 02:49:46.526077 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09cc6f56d72b49448472ca156cd705fe" candidate_uuid: "75a99e4eb38b443e8ec6770ab3425739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4a5220ff004058a1e0d067c4db4b7e"
I20260619 02:49:46.526075 24327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09cc6f56d72b49448472ca156cd705fe" candidate_uuid: "75a99e4eb38b443e8ec6770ab3425739" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c580b8d4554aa38b61570f591243e9"
I20260619 02:49:46.526149 24047 raft_consensus.cc:3037] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:46.526152 24327 raft_consensus.cc:3037] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:46.526713 24047 raft_consensus.cc:2454] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75a99e4eb38b443e8ec6770ab3425739 in term 1.
I20260619 02:49:46.526836 24327 raft_consensus.cc:2454] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75a99e4eb38b443e8ec6770ab3425739 in term 1.
I20260619 02:49:46.526894 24119 leader_election.cc:302] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4a5220ff004058a1e0d067c4db4b7e, 75a99e4eb38b443e8ec6770ab3425739; no voters:
I20260619 02:49:46.526981 24490 raft_consensus.cc:2781] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:46.527122 24490 raft_consensus.cc:686] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 1 LEADER]: Becoming Leader. State: Replica: 75a99e4eb38b443e8ec6770ab3425739, State: Running, Role: LEADER
I20260619 02:49:46.527194 24490 consensus_queue.cc:237] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } }
I20260619 02:49:46.527993 23890 catalog_manager.cc:5495] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194). New cstate: current_term: 1 leader_uuid: "75a99e4eb38b443e8ec6770ab3425739" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75a99e4eb38b443e8ec6770ab3425739" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20260619 02:49:46.546576 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.544079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.21.164.194:45587)
0619 02:49:46.544244 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:46.544246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:46.544251 (+ 5us) server_negotiation.cc:407] Connection header received
0619 02:49:46.544341 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:46.544342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:46.544367 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:46.544446 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:46.544572 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.545107 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.545794 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.546034 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.546088 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.546238 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:46.546356 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:46.546419 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:46.546464 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 02:49:46.546483 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
I20260619 02:49:46.551759 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.549056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:43555 (local address 127.21.164.195:33457)
0619 02:49:46.549102 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:46.549104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:46.549163 (+ 59us) server_negotiation.cc:407] Connection header received
0619 02:49:46.549224 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:46.549225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:46.549235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:46.549282 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:46.549401 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.549863 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.550575 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.550760 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.550816 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.551400 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:46.551431 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:46.551568 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:46.551650 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 02:49:46.551676 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:49:46.551784 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.548945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:33457 (local address 127.21.164.194:43555)
0619 02:49:46.549112 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.549123 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:46.549178 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:46.549288 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:46.549288 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:46.549290 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:46.549350 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.549351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.549927 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.549927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.550505 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.550507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.550829 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.550830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.551312 (+ 482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.551325 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:46.551590 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:46.551593 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:46.551635 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:49:46.551683 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260619 02:49:46.553712 24047 raft_consensus.cc:1261] T 09cc6f56d72b49448472ca156cd705fe P 1d4a5220ff004058a1e0d067c4db4b7e [term 1 FOLLOWER]: Refusing update from remote peer 75a99e4eb38b443e8ec6770ab3425739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:46.553730 24327 raft_consensus.cc:1261] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Refusing update from remote peer 75a99e4eb38b443e8ec6770ab3425739: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:46.553947 24490 consensus_queue.cc:1035] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4a5220ff004058a1e0d067c4db4b7e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:46.554081 24503 consensus_queue.cc:1035] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c580b8d4554aa38b61570f591243e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:46.560804 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.558341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:45587 (local address 127.21.164.195:44495)
0619 02:49:46.558465 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:46.558474 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:46.558525 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:46.558642 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:46.558643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:46.558644 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:49:46.558691 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.558693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.559381 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.559383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.559977 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:46.559980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.560324 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.560325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:46.560377 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.560410 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:49:46.560616 (+ 206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:49:46.560618 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:49:46.560682 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:49:46.560727 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":83,"threads_started":1}
I20260619 02:49:46.560796 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:46.558375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:44495 (local address 127.21.164.194:45587)
0619 02:49:46.558429 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:46.558431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:46.558525 (+ 94us) server_negotiation.cc:407] Connection header received
0619 02:49:46.558580 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:46.558581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:46.558591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:46.558639 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:46.558735 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.559314 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.560047 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:46.560274 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:46.560347 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:46.560481 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:46.560513 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:46.560554 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:46.560695 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 02:49:46.560722 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 02:49:46.561072 24455 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:49:48.078158 23888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:48.078290 23888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:48.079178 23888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486e66c8d95148bf973aa60caa49ef1f: Sending DeleteTablet for 3 replicas of tablet 09cc6f56d72b49448472ca156cd705fe
I20260619 02:49:48.079429 24167 tablet_service.cc:1514] Processing DeleteTablet for tablet 09cc6f56d72b49448472ca156cd705fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:53200
I20260619 02:49:48.079438 24307 tablet_service.cc:1514] Processing DeleteTablet for tablet 09cc6f56d72b49448472ca156cd705fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:43720
I20260619 02:49:48.079453 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 09cc6f56d72b49448472ca156cd705fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:48 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:34276
I20260619 02:49:48.079545 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 23962
I20260619 02:49:48.079658 24539 tablet_replica.cc:330] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9: stopping tablet replica
I20260619 02:49:48.080331 24539 raft_consensus.cc:2229] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:48.080379 24539 raft_consensus.cc:2258] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:48.080483 24537 tablet_replica.cc:330] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739: stopping tablet replica
I20260619 02:49:48.080538 24537 raft_consensus.cc:2229] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:48.080636 24537 raft_consensus.cc:2258] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:48.080888 24539 ts_tablet_manager.cc:1905] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:48.081072 24537 ts_tablet_manager.cc:1905] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:48.083036 24539 ts_tablet_manager.cc:1918] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:49:48.083036 24537 ts_tablet_manager.cc:1918] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:49:48.083096 24539 log.cc:1196] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/09cc6f56d72b49448472ca156cd705fe
I20260619 02:49:48.083096 24537 log.cc:1196] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/09cc6f56d72b49448472ca156cd705fe
I20260619 02:49:48.083304 24539 ts_tablet_manager.cc:1939] T 09cc6f56d72b49448472ca156cd705fe P 74c580b8d4554aa38b61570f591243e9: Deleting consensus metadata
I20260619 02:49:48.083349 24537 ts_tablet_manager.cc:1939] T 09cc6f56d72b49448472ca156cd705fe P 75a99e4eb38b443e8ec6770ab3425739: Deleting consensus metadata
I20260619 02:49:48.083592 23877 catalog_manager.cc:4841] TS 74c580b8d4554aa38b61570f591243e9 (127.21.164.195:33457): tablet 09cc6f56d72b49448472ca156cd705fe (table test-table [id=1f0f740e077645068a46b2da4fb85cb1]) successfully deleted
I20260619 02:49:48.083664 23877 catalog_manager.cc:4841] TS 75a99e4eb38b443e8ec6770ab3425739 (127.21.164.194:45587): tablet 09cc6f56d72b49448472ca156cd705fe (table test-table [id=1f0f740e077645068a46b2da4fb85cb1]) successfully deleted
W20260619 02:49:48.086182 23875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:44941 (error 108)
I20260619 02:49:48.086412 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 24102
I20260619 02:49:48.088858 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:48.086512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:44941 (local address 127.0.0.1:34302)
0619 02:49:48.088752 (+ 2240us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:48.088812 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:44941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2203,"thread_start_us":56,"threads_started":1}
W20260619 02:49:48.089020 23875 catalog_manager.cc:4584] TS 1d4a5220ff004058a1e0d067c4db4b7e (127.21.164.193:44941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09cc6f56d72b49448472ca156cd705fe: Network error: Client connection negotiation failed: client connection to 127.21.164.193:44941: connect: Connection refused (error 111)
I20260619 02:49:48.091513 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 24242
I20260619 02:49:48.096319 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 23858
2026-06-19T02:49:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5909 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:48 dist-test-slave-jt0t krb5kdc[24544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:48 dist-test-slave-jt0t krb5kdc[24544](info): set up 2 sockets
Jun 19 02:49:48 dist-test-slave-jt0t krb5kdc[24544](info): commencing operation
krb5kdc: starting...
W20260619 02:49:49.036165 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[24544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:49:49.150818 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42903
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:42903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:49.230806 24557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:49.231030 24557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:49.231060 24557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:49.232266 24557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:49.232312 24557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:49.232329 24557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:49.232347 24557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:49.232362 24557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:49.233841 24557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:42903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:49.234088 24557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:49.234302 24557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:49.239046 24562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:49.239048 24563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:49.239048 24565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:49.239318 24557 server_base.cc:1034] running on GCE node
I20260619 02:49:49.239547 24557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:49.239831 24557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:49.240967 24557 hybrid_clock.cc:648] HybridClock initialized: now 1781837389240944 us; error 40 us; skew 500 ppm
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[24544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:49.250532 24557 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:49:49.251060 24557 webserver.cc:460] Webserver started at http://127.21.164.254:38239/ using document root <none> and password file <none>
I20260619 02:49:49.251312 24557 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:49.251367 24557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:49.251497 24557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:49.252591 24557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "885cadac92c0414e8f5c176c1858f2c9"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "fb61774a4d32ee2e077eb9f2274e2568"
server_key_iv: "e6c7671fcb8e698b62daf733a4430145"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.252982 24557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "885cadac92c0414e8f5c176c1858f2c9"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "fb61774a4d32ee2e077eb9f2274e2568"
server_key_iv: "e6c7671fcb8e698b62daf733a4430145"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.254283 24557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:49.255050 24572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.255268 24557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:49.255354 24557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "885cadac92c0414e8f5c176c1858f2c9"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "fb61774a4d32ee2e077eb9f2274e2568"
server_key_iv: "e6c7671fcb8e698b62daf733a4430145"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.255419 24557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:49.277323 24557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:49.279515 24557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:49.279661 24557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:49.283420 24557 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:42903
I20260619 02:49:49.283460 24624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:42903 every 8 connection(s)
I20260619 02:49:49.283788 24557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:49.284703 24625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:49.285722 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 24557
I20260619 02:49:49.285830 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:49.286000 22163 external_mini_cluster.cc:1428] Setting key d14b5d606718c4042d5493d80d640f42
I20260619 02:49:49.287384 24625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9: Bootstrap starting.
I20260619 02:49:49.288093 24625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:49.288360 24625 log.cc:824] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:49.289047 24625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9: No bootstrap required, opened a new log
I20260619 02:49:49.291373 24625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885cadac92c0414e8f5c176c1858f2c9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42903 } }
I20260619 02:49:49.291553 24625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:49.291589 24625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885cadac92c0414e8f5c176c1858f2c9, State: Initialized, Role: FOLLOWER
I20260619 02:49:49.291728 24625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885cadac92c0414e8f5c176c1858f2c9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42903 } }
I20260619 02:49:49.291807 24625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:49.291843 24625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:49.291893 24625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:49.292528 24625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885cadac92c0414e8f5c176c1858f2c9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42903 } }
I20260619 02:49:49.292618 24625 leader_election.cc:302] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 885cadac92c0414e8f5c176c1858f2c9; no voters:
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[24544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:49.292800 24625 leader_election.cc:288] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:49.292826 24630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:49.292941 24630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [term 1 LEADER]: Becoming Leader. State: Replica: 885cadac92c0414e8f5c176c1858f2c9, State: Running, Role: LEADER
I20260619 02:49:49.293021 24630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885cadac92c0414e8f5c176c1858f2c9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42903 } }
I20260619 02:49:49.293051 24625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:49.293380 24630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 885cadac92c0414e8f5c176c1858f2c9. Latest consensus state: current_term: 1 leader_uuid: "885cadac92c0414e8f5c176c1858f2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885cadac92c0414e8f5c176c1858f2c9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42903 } } }
I20260619 02:49:49.293447 24630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:49.293727 24631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "885cadac92c0414e8f5c176c1858f2c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885cadac92c0414e8f5c176c1858f2c9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42903 } } }
I20260619 02:49:49.294188 24631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:49.294368 24635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:49.294853 24635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:49.296357 24635 catalog_manager.cc:1347] Generated new cluster ID: 42fc9d5982334c3e8592545daf5e5732
I20260619 02:49:49.296408 24635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:49.296995 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:49.286513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.21.164.254:42903)
0619 02:49:49.286667 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:49.286676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:49.286703 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:49:49.287191 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:49.287198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:49.287245 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:49.287358 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:49.288095 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.289196 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.290210 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.290605 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.290679 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:49.293330 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:49.293334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:49.293335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:49.293349 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:49.295243 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:49.295632 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:49.295635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:49.295635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:49.295736 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:49.295951 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:49.295952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:49.295952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:49.296181 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:49.296318 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:49.296745 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 02:49:49.296858 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260619 02:49:49.310863 24635 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:49.311597 24635 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:49.317577 24635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 885cadac92c0414e8f5c176c1858f2c9: Generated new TSK 0
I20260619 02:49:49.317791 24635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:49:49.349440 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42903
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:49.429459 24651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:49.429682 24651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:49.429713 24651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:49.430876 24651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:49.430922 24651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:49.430958 24651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:49:49.432367 24651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:49.432606 24651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:49.432838 24651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:49.437467 24656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:49.437470 24659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:49.437459 24657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:49.437636 24651 server_base.cc:1034] running on GCE node
I20260619 02:49:49.437904 24651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:49.438133 24651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:49.439271 24651 hybrid_clock.cc:648] HybridClock initialized: now 1781837389439256 us; error 27 us; skew 500 ppm
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[24544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:49.441700 24651 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:49:49.442128 24651 webserver.cc:460] Webserver started at http://127.21.164.193:45313/ using document root <none> and password file <none>
I20260619 02:49:49.442334 24651 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:49.442385 24651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:49.442507 24651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:49.443516 24651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "e0d7b86f40c8470ba5ad3e6a399becc0"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "4130ff0784441a5ff153fcc40ed03a80"
server_key_iv: "4228bc0197c4c787e84b65cf13496a8d"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.443842 24651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "e0d7b86f40c8470ba5ad3e6a399becc0"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "4130ff0784441a5ff153fcc40ed03a80"
server_key_iv: "4228bc0197c4c787e84b65cf13496a8d"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.445123 24651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:49.445741 24666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.445878 24651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:49.445956 24651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "e0d7b86f40c8470ba5ad3e6a399becc0"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "4130ff0784441a5ff153fcc40ed03a80"
server_key_iv: "4228bc0197c4c787e84b65cf13496a8d"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.446014 24651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:49.467481 24651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:49.469703 24651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:49.469843 24651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:49.470064 24651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:49.470356 24651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:49.470389 24651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.470429 24651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:49.470453 24651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.475719 24651 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:34825
I20260619 02:49:49.475783 24779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:34825 every 8 connection(s)
I20260619 02:49:49.476068 24651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[24544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:49.484468 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:49.476777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:58913 (local address 127.21.164.254:42903)
0619 02:49:49.476845 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:49.476847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:49.477090 (+ 243us) server_negotiation.cc:407] Connection header received
0619 02:49:49.477605 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:49.477606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:49.477617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:49.477667 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:49.478562 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.479067 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.480023 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.480236 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.480291 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:49.482310 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:49.482311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:49.482312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:49.482326 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:49.483386 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:49.483723 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:49.483724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:49.483724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:49.483767 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:49.483947 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:49.483948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:49.483948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:49.484041 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:49.484083 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:49.484286 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 02:49:49.484382 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:49:49.484575 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:49.476777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42903 (local address 127.21.164.193:58913)
0619 02:49:49.476971 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:49.477009 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:49.477520 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:49.477723 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:49.477726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:49.478274 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:49.478470 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:49.478474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.479133 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.479134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:49.479937 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:49.479939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.480297 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.480297 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:49.480391 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:49.480935 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:49.480943 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:49.482233 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:49.483455 (+ 1222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:49.483461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:49.483464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:49.483669 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:49.483810 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:49.483810 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:49.483810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:49.483904 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:49.484081 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:49.484084 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:49.484194 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 02:49:49.484271 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:49:49.484387 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":119,"threads_started":1}
I20260619 02:49:49.484638 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 24651
I20260619 02:49:49.484743 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:49.484912 22163 external_mini_cluster.cc:1428] Setting key 6b1ad52dae6e3075db79d6ee24fa10aa
I20260619 02:49:49.485232 24780 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42903
I20260619 02:49:49.485329 24780 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:49.485507 24780 heartbeater.cc:511] Master 127.21.164.254:42903 requested a full tablet report, sending...
I20260619 02:49:49.486032 24589 ts_manager.cc:194] Registered new tserver with Master: e0d7b86f40c8470ba5ad3e6a399becc0 (127.21.164.193:34825)
I20260619 02:49:49.486635 24589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:58913
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:49:49.507581 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42903
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:49.586828 24785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:49.587041 24785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:49.587070 24785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:49.588227 24785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:49.588275 24785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:49.588311 24785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:49:49.589762 24785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:49.589998 24785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:49.590204 24785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:49.595316 24791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:49.595351 24790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:49.595312 24793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:49.595523 24785 server_base.cc:1034] running on GCE node
I20260619 02:49:49.595707 24785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:49.595913 24785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:49.597054 24785 hybrid_clock.cc:648] HybridClock initialized: now 1781837389597042 us; error 26 us; skew 500 ppm
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[24544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:49.599449 24785 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:49:49.599891 24785 webserver.cc:460] Webserver started at http://127.21.164.194:43395/ using document root <none> and password file <none>
I20260619 02:49:49.600092 24785 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:49.600140 24785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:49.600266 24785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:49.601356 24785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "175457803c40480da6d78c8df5075057"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "81153a1d7481a8a388afc99922d8cf31"
server_key_iv: "ad70056eed7e736f2bce8daaa1cf2e35"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.601697 24785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "175457803c40480da6d78c8df5075057"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "81153a1d7481a8a388afc99922d8cf31"
server_key_iv: "ad70056eed7e736f2bce8daaa1cf2e35"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.602934 24785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:49.603622 24800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.603768 24785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:49.603848 24785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "175457803c40480da6d78c8df5075057"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "81153a1d7481a8a388afc99922d8cf31"
server_key_iv: "ad70056eed7e736f2bce8daaa1cf2e35"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.603909 24785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:49.620982 24785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:49.623193 24785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:49.623332 24785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:49.623561 24785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:49.623855 24785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:49.623888 24785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.623919 24785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:49.623935 24785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.629300 24785 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:34493
I20260619 02:49:49.629350 24913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:34493 every 8 connection(s)
I20260619 02:49:49.629649 24785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:49.632105 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 24785
I20260619 02:49:49.632182 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:49.632345 22163 external_mini_cluster.cc:1428] Setting key ab3f10375eab8289a285e3b308f2e51b
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[24544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:49.639048 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:49.630297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:48407 (local address 127.21.164.254:42903)
0619 02:49:49.630344 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:49.630346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:49.630531 (+ 185us) server_negotiation.cc:407] Connection header received
0619 02:49:49.631005 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:49.631006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:49.631015 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:49.631056 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:49.631944 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.632506 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.633635 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.633901 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.633964 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:49.636357 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:49.636359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:49.636359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:49.636366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:49.637660 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:49.638146 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:49.638146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:49.638146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:49.638193 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:49.638453 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:49.638453 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:49.638454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:49.638550 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:49.638594 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:49.638923 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 02:49:49.638982 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 02:49:49.639317 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:49.630268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42903 (local address 127.21.164.194:48407)
0619 02:49:49.630420 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:49.630462 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:49.630909 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:49.631114 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:49.631117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:49.631677 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:49.631869 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:49.631874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.632578 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.632579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:49.633544 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:49.633547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.633975 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.633976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:49.634071 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:49.634570 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:49.634580 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:49.636265 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:49.637744 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:49.637751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:49.637754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:49.638074 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:49.638249 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:49.638250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:49.638250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:49.638395 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:49.638602 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:49.638607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:49.638783 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 02:49:49.638909 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 02:49:49.639081 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260619 02:49:49.639843 24914 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42903
I20260619 02:49:49.639935 24914 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:49.640107 24914 heartbeater.cc:511] Master 127.21.164.254:42903 requested a full tablet report, sending...
I20260619 02:49:49.640504 24589 ts_manager.cc:194] Registered new tserver with Master: 175457803c40480da6d78c8df5075057 (127.21.164.194:34493)
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
I20260619 02:49:49.640944 24589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:48407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:49:49.657538 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42903
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:49.737062 24919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:49.737264 24919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:49.737286 24919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:49.738444 24919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:49.738485 24919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:49.738512 24919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:49:49.739930 24919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:49.740145 24919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:49.740388 24919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:49.745052 24925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:49.745114 24919 server_base.cc:1034] running on GCE node
W20260619 02:49:49.745047 24924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:49.745048 24927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:49.745452 24919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:49.745648 24919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:49.746780 24919 hybrid_clock.cc:648] HybridClock initialized: now 1781837389746770 us; error 26 us; skew 500 ppm
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[24544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:49.749181 24919 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:49:49.749610 24919 webserver.cc:460] Webserver started at http://127.21.164.195:44263/ using document root <none> and password file <none>
I20260619 02:49:49.749850 24919 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:49.749898 24919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:49.750021 24919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:49.751030 24919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "9c04eff8834d434b83dcd75e484f8668"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "72aec5a606ae55ef97c3ef235a9c6d08"
server_key_iv: "8629212e3dbc99d914b30f890f23e83b"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.751368 24919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "9c04eff8834d434b83dcd75e484f8668"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "72aec5a606ae55ef97c3ef235a9c6d08"
server_key_iv: "8629212e3dbc99d914b30f890f23e83b"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.752593 24919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 02:49:49.753257 24934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.753419 24919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:49.753499 24919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "9c04eff8834d434b83dcd75e484f8668"
format_stamp: "Formatted at 2026-06-19 02:49:49 on dist-test-slave-jt0t"
server_key: "72aec5a606ae55ef97c3ef235a9c6d08"
server_key_iv: "8629212e3dbc99d914b30f890f23e83b"
server_key_version: "encryptionkey@0"
I20260619 02:49:49.753557 24919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:49.801998 24919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:49.803913 24919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:49.804045 24919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:49.804250 24919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:49.804528 24919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:49.804561 24919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.804584 24919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:49.804606 24919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:49.810055 24919 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:32833
I20260619 02:49:49.810123 25047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:32833 every 8 connection(s)
I20260619 02:49:49.810401 24919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:49.812767 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 24919
I20260619 02:49:49.812860 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:49.813035 22163 external_mini_cluster.cc:1428] Setting key 5884ef8c2c847fc5bde9c50970b64722
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[24544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837389, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:49.819576 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:49.811065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:39539 (local address 127.21.164.254:42903)
0619 02:49:49.811119 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:49.811120 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:49.811378 (+ 258us) server_negotiation.cc:407] Connection header received
0619 02:49:49.811868 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:49.811870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:49.811881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:49.811948 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:49.812991 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.813692 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.814674 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.814862 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.814931 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:49.817139 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:49.817140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:49.817140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:49.817147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:49.818494 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:49.818864 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:49.818865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:49.818865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:49.818912 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:49.819092 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:49.819092 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:49.819092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:49.819188 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:49.819232 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:49.819449 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 02:49:49.819508 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:49:49.819768 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:49.811033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42903 (local address 127.21.164.195:39539)
0619 02:49:49.811250 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:49.811292 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:49.811761 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:49.812002 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:49.812005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:49.812575 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:49.812891 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:49.812898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.813763 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.813764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:49.814596 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:49.814599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:49.815013 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:49.815015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:49.815127 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:49.815667 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:49.815676 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:49.817051 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:49.818569 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:49.818577 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:49.818579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:49.818806 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:49.818956 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:49.818957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:49.818957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:49.819052 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:49.819234 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:49.819237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:49.819355 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 02:49:49.819431 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:49:49.819551 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260619 02:49:49.820209 25048 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42903
I20260619 02:49:49.820305 25048 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:49.820456 25048 heartbeater.cc:511] Master 127.21.164.254:42903 requested a full tablet report, sending...
I20260619 02:49:49.820822 24589 ts_manager.cc:194] Registered new tserver with Master: 9c04eff8834d434b83dcd75e484f8668 (127.21.164.195:32833)
I20260619 02:49:49.821161 24589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:39539
I20260619 02:49:49.824754 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:49:49.834862 24628 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:49.833999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55028 (local address 127.21.164.254:42903)
0619 02:49:49.834055 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:49.834057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:49.834168 (+ 111us) server_negotiation.cc:407] Connection header received
0619 02:49:49.834220 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:49.834221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:49.834229 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:49.834278 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:49.834789 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55028: BlockingRecv error: recv got EOF from 127.0.0.1:55028 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:49:49.835661 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 24651
I20260619 02:49:49.839363 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 24785
I20260619 02:49:49.842912 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 24919
I20260619 02:49:49.846554 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 24557
2026-06-19T02:49:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1749 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[25060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[25060](info): set up 2 sockets
Jun 19 02:49:49 dist-test-slave-jt0t krb5kdc[25060](info): commencing operation
krb5kdc: starting...
W20260619 02:49:50.788242 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:49:50 dist-test-slave-jt0t krb5kdc[25060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:49:50.901204 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:43415
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:43415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:50.981455 25073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:50.981670 25073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:50.981691 25073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:50.982887 25073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:50.982930 25073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:50.982944 25073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:50.982955 25073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:50.982966 25073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:50.984513 25073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:43415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:43415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:50.984778 25073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:50.984992 25073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:50.989739 25078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:50.989751 25081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:50.989763 25079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:50.989956 25073 server_base.cc:1034] running on GCE node
I20260619 02:49:50.990197 25073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:50.990469 25073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:50.991608 25073 hybrid_clock.cc:648] HybridClock initialized: now 1781837390991589 us; error 34 us; skew 500 ppm
Jun 19 02:49:50 dist-test-slave-jt0t krb5kdc[25060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837390, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:50.993913 25073 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:49:50.994323 25073 webserver.cc:460] Webserver started at http://127.21.164.254:33255/ using document root <none> and password file <none>
I20260619 02:49:50.994518 25073 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:50.994561 25073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:50.994678 25073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:50.995687 25073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "9eb9d55145fa4690acbc420409b3673c"
format_stamp: "Formatted at 2026-06-19 02:49:50 on dist-test-slave-jt0t"
server_key: "67a5b6df30fd93c0cc8b7f0df34f14f6"
server_key_iv: "b07903400a4a8ed4c4034b343f7872fa"
server_key_version: "encryptionkey@0"
I20260619 02:49:50.996007 25073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "9eb9d55145fa4690acbc420409b3673c"
format_stamp: "Formatted at 2026-06-19 02:49:50 on dist-test-slave-jt0t"
server_key: "67a5b6df30fd93c0cc8b7f0df34f14f6"
server_key_iv: "b07903400a4a8ed4c4034b343f7872fa"
server_key_version: "encryptionkey@0"
I20260619 02:49:50.997279 25073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:50.997920 25088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:50.998070 25073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:50.998152 25073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "9eb9d55145fa4690acbc420409b3673c"
format_stamp: "Formatted at 2026-06-19 02:49:50 on dist-test-slave-jt0t"
server_key: "67a5b6df30fd93c0cc8b7f0df34f14f6"
server_key_iv: "b07903400a4a8ed4c4034b343f7872fa"
server_key_version: "encryptionkey@0"
I20260619 02:49:50.998210 25073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:51.015013 25073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:51.016855 25073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:51.017002 25073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:51.020753 25073 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:43415
I20260619 02:49:51.020797 25140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:43415 every 8 connection(s)
I20260619 02:49:51.021281 25073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:51.021941 25141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:51.024155 25141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c: Bootstrap starting.
I20260619 02:49:51.024701 25141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:51.024931 25141 log.cc:824] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:51.025508 25141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c: No bootstrap required, opened a new log
I20260619 02:49:51.025796 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 25073
I20260619 02:49:51.025878 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:51.026034 22163 external_mini_cluster.cc:1428] Setting key 4d8f9cf51ad7b9eae6a15527d9653edc
I20260619 02:49:51.027899 25141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb9d55145fa4690acbc420409b3673c" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43415 } }
I20260619 02:49:51.028080 25141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:51.028115 25141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb9d55145fa4690acbc420409b3673c, State: Initialized, Role: FOLLOWER
I20260619 02:49:51.028270 25141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb9d55145fa4690acbc420409b3673c" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43415 } }
I20260619 02:49:51.028342 25141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:51.028364 25141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:51.028406 25141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:51.029085 25141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb9d55145fa4690acbc420409b3673c" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43415 } }
I20260619 02:49:51.029188 25141 leader_election.cc:302] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eb9d55145fa4690acbc420409b3673c; no voters:
I20260619 02:49:51.029345 25141 leader_election.cc:288] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:51.029402 25146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:51.029573 25146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [term 1 LEADER]: Becoming Leader. State: Replica: 9eb9d55145fa4690acbc420409b3673c, State: Running, Role: LEADER
I20260619 02:49:51.029580 25141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:51.029722 25146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb9d55145fa4690acbc420409b3673c" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43415 } }
I20260619 02:49:51.030107 25148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eb9d55145fa4690acbc420409b3673c. Latest consensus state: current_term: 1 leader_uuid: "9eb9d55145fa4690acbc420409b3673c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb9d55145fa4690acbc420409b3673c" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43415 } } }
I20260619 02:49:51.030104 25147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eb9d55145fa4690acbc420409b3673c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eb9d55145fa4690acbc420409b3673c" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43415 } } }
I20260619 02:49:51.030216 25148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:51.030226 25147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:51.031064 25155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:51.031668 25155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 02:49:51 dist-test-slave-jt0t krb5kdc[25060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:51.034492 25155 catalog_manager.cc:1347] Generated new cluster ID: ac27b0914b5c42c7aa0b829e9b6a66e8
I20260619 02:49:51.034545 25155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:51.038288 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:51.026483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.21.164.254:43415)
0619 02:49:51.026683 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:51.026693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:51.026726 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:49:51.027873 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:51.027881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:51.027958 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:51.028105 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:51.028814 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:51.030181 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:51.031222 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:51.031769 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:51.031872 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:51.034501 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:51.034505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:51.034506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:51.034518 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:51.036378 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:51.036778 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:51.036780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:51.036781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:51.036892 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:51.037072 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:51.037072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:51.037073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:51.037364 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:51.037517 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:51.037941 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 02:49:51.038083 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260619 02:49:51.039184 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:51.049243 25155 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:51.050007 25155 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:51.059484 25155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eb9d55145fa4690acbc420409b3673c: Generated new TSK 0
I20260619 02:49:51.059701 25155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:51.061239 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:51.057715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.21.164.254:43415)
0619 02:49:51.057762 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:51.057764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:51.057883 (+ 119us) server_negotiation.cc:407] Connection header received
0619 02:49:51.057961 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:51.057962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:51.057975 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:51.058022 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:51.058409 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:51.058909 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:51.059841 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:51.060041 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:51.060096 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:51.060810 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:51.060811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:51.060812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:51.061007 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:51.061025 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:51.061032 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:51.061079 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:51.061142 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 02:49:51.061168 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:49:51.063458 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 25073
2026-06-19T02:49:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1215 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:51 dist-test-slave-jt0t krb5kdc[25174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:51 dist-test-slave-jt0t krb5kdc[25174](info): set up 2 sockets
Jun 19 02:49:51 dist-test-slave-jt0t krb5kdc[25174](info): commencing operation
krb5kdc: starting...
W20260619 02:49:52.004339 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:49:52.117008 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:44495
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:44495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:52.196049 25187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:52.196254 25187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:52.196276 25187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:52.197444 25187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:52.197484 25187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:52.197496 25187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:52.197508 25187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:52.197532 25187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:52.198989 25187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:44495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:44495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:52.199219 25187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:52.199429 25187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:52.204198 25192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.204193 25193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:52.204277 25187 server_base.cc:1034] running on GCE node
W20260619 02:49:52.204193 25195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:52.204620 25187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:52.204937 25187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:52.206097 25187 hybrid_clock.cc:648] HybridClock initialized: now 1781837392206081 us; error 28 us; skew 500 ppm
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:52.208498 25187 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:49:52.208942 25187 webserver.cc:460] Webserver started at http://127.21.164.254:43539/ using document root <none> and password file <none>
I20260619 02:49:52.209131 25187 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:52.209169 25187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:52.209259 25187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:52.210275 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "5a10662f2e3f48849b1d4efccf0d24cb"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "942759f06bddce6118e50aac57ab18ad"
server_key_iv: "a506bf465e6fa5f57e4c346ce1174969"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.210572 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "5a10662f2e3f48849b1d4efccf0d24cb"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "942759f06bddce6118e50aac57ab18ad"
server_key_iv: "a506bf465e6fa5f57e4c346ce1174969"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.211747 25187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 02:49:52.212364 25202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.212538 25187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:52.212620 25187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "5a10662f2e3f48849b1d4efccf0d24cb"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "942759f06bddce6118e50aac57ab18ad"
server_key_iv: "a506bf465e6fa5f57e4c346ce1174969"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.212705 25187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:52.229723 25187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:52.231890 25187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:52.232008 25187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:52.236006 25187 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:44495
I20260619 02:49:52.236044 25254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:44495 every 8 connection(s)
I20260619 02:49:52.236357 25187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:52.237310 25255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:52.239496 25255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb: Bootstrap starting.
I20260619 02:49:52.240032 25255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:52.240273 25255 log.cc:824] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:52.240896 25255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb: No bootstrap required, opened a new log
I20260619 02:49:52.241613 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 25187
I20260619 02:49:52.241717 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:52.241909 22163 external_mini_cluster.cc:1428] Setting key be0d73da41f7e44b32cf20867d813287
I20260619 02:49:52.242812 25255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10662f2e3f48849b1d4efccf0d24cb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 44495 } }
I20260619 02:49:52.242955 25255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:52.242986 25255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a10662f2e3f48849b1d4efccf0d24cb, State: Initialized, Role: FOLLOWER
I20260619 02:49:52.243124 25255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10662f2e3f48849b1d4efccf0d24cb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 44495 } }
I20260619 02:49:52.243196 25255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:52.243227 25255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:52.243273 25255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:52.244059 25255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10662f2e3f48849b1d4efccf0d24cb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 44495 } }
I20260619 02:49:52.244151 25255 leader_election.cc:302] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a10662f2e3f48849b1d4efccf0d24cb; no voters:
I20260619 02:49:52.244302 25255 leader_election.cc:288] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:52.244333 25260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:52.244442 25260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [term 1 LEADER]: Becoming Leader. State: Replica: 5a10662f2e3f48849b1d4efccf0d24cb, State: Running, Role: LEADER
I20260619 02:49:52.244546 25255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:52.244576 25260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10662f2e3f48849b1d4efccf0d24cb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 44495 } }
I20260619 02:49:52.244979 25260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a10662f2e3f48849b1d4efccf0d24cb. Latest consensus state: current_term: 1 leader_uuid: "5a10662f2e3f48849b1d4efccf0d24cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10662f2e3f48849b1d4efccf0d24cb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 44495 } } }
I20260619 02:49:52.245102 25260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:52.245436 25269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:52.245991 25269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:52.246763 25261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a10662f2e3f48849b1d4efccf0d24cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a10662f2e3f48849b1d4efccf0d24cb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 44495 } } }
I20260619 02:49:52.247483 25269 catalog_manager.cc:1347] Generated new cluster ID: f8f821802388441abf32e5a9c501d254
I20260619 02:49:52.247982 25261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:52.248013 25269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:52.253190 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.242403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.21.164.254:44495)
0619 02:49:52.242580 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.242588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.242612 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:49:52.242775 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.242780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.242822 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.242937 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.243592 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.244576 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.245523 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.246120 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.246221 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.249169 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:52.249173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:52.249174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:52.249185 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:52.251075 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.251479 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.251482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.251482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.252030 (+ 548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.252244 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.252245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.252246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.252543 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:52.252711 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:52.252843 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:49:52.252976 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260619 02:49:52.262673 25269 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:52.263195 25269 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:52.271459 25269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a10662f2e3f48849b1d4efccf0d24cb: Generated new TSK 0
I20260619 02:49:52.271656 25269 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:49:52.302062 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:44495
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:52.401549 25281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:52.401744 25281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:52.401767 25281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:52.402874 25281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:52.402916 25281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:52.402943 25281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:49:52.404345 25281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:44495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:52.404604 25281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:52.404841 25281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:52.409380 25289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.409387 25286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.409675 25287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:52.409789 25281 server_base.cc:1034] running on GCE node
I20260619 02:49:52.409992 25281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:52.410213 25281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:52.411345 25281 hybrid_clock.cc:648] HybridClock initialized: now 1781837392411330 us; error 27 us; skew 500 ppm
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:52.413738 25281 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:49:52.414175 25281 webserver.cc:460] Webserver started at http://127.21.164.193:41013/ using document root <none> and password file <none>
I20260619 02:49:52.414378 25281 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:52.414426 25281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:52.414551 25281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:52.415594 25281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "3332b9976dd54db2b8f5edb781ddb9ff"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "e0ffb498dd012c4d74793fef2864e1f2"
server_key_iv: "cafee0cdcf0aa18a3f2cafb465c6993f"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.415922 25281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "3332b9976dd54db2b8f5edb781ddb9ff"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "e0ffb498dd012c4d74793fef2864e1f2"
server_key_iv: "cafee0cdcf0aa18a3f2cafb465c6993f"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.417217 25281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:52.417862 25296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.418002 25281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:52.418084 25281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "3332b9976dd54db2b8f5edb781ddb9ff"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "e0ffb498dd012c4d74793fef2864e1f2"
server_key_iv: "cafee0cdcf0aa18a3f2cafb465c6993f"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.418143 25281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:52.434873 25281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:52.437108 25281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:52.437248 25281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:52.437465 25281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:52.437757 25281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:52.437794 25281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.437824 25281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:52.437845 25281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.443113 25281 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:44989
I20260619 02:49:52.443174 25409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:44989 every 8 connection(s)
I20260619 02:49:52.443462 25281 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:52.447602 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 25281
I20260619 02:49:52.447693 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:52.447868 22163 external_mini_cluster.cc:1428] Setting key cad59eb2f72b06675e5315c5024ecbd8
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:52.453236 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.444140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:53353 (local address 127.21.164.254:44495)
0619 02:49:52.444210 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.444212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.444383 (+ 171us) server_negotiation.cc:407] Connection header received
0619 02:49:52.444958 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.444959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.444976 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.445039 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.446235 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.446769 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.447831 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.448131 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.448203 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.450603 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:52.450605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:52.450605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:52.450616 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:52.451928 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.452318 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.452319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.452319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.452372 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.452586 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.452587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.452588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.452744 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:52.452802 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:52.453060 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 02:49:52.453144 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:49:52.453413 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.444116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:44495 (local address 127.21.164.193:53353)
0619 02:49:52.444259 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:52.444301 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:52.444814 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:52.445102 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:52.445106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:52.445872 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:52.446140 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.446145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.446841 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.446842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.447673 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.447676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.448219 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.448220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.448390 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.448986 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:52.448996 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:52.450510 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:52.451998 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:52.452005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:52.452008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:52.452227 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:52.452426 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:52.452427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:52.452427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:52.452533 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:52.452799 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:52.452805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:52.452943 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 02:49:52.453033 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:49:52.453190 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260619 02:49:52.453872 25410 heartbeater.cc:347] Connected to a master server at 127.21.164.254:44495
I20260619 02:49:52.453963 25410 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:52.454135 25410 heartbeater.cc:511] Master 127.21.164.254:44495 requested a full tablet report, sending...
I20260619 02:49:52.454658 25219 ts_manager.cc:194] Registered new tserver with Master: 3332b9976dd54db2b8f5edb781ddb9ff (127.21.164.193:44989)
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
I20260619 02:49:52.455268 25219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:53353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:49:52.471854 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:44495
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:52.555008 25415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:52.555226 25415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:52.555256 25415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:52.556412 25415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:52.556453 25415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:52.556489 25415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:49:52.557945 25415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:44495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:52.558176 25415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:52.558383 25415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:52.563158 25423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.563171 25421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.563244 25420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:52.563356 25415 server_base.cc:1034] running on GCE node
I20260619 02:49:52.563578 25415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:52.563797 25415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:52.564939 25415 hybrid_clock.cc:648] HybridClock initialized: now 1781837392564923 us; error 28 us; skew 500 ppm
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:52.567395 25415 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:49:52.567838 25415 webserver.cc:460] Webserver started at http://127.21.164.194:39839/ using document root <none> and password file <none>
I20260619 02:49:52.568042 25415 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:52.568089 25415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:52.568212 25415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:52.569265 25415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "db766a40a8fe42e3ba4cd565c234b9a3"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "efdeb3ba0b2b035861e6acf82614dee0"
server_key_iv: "1c0b9ce9311e1794486de772d44de4ec"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.569617 25415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "db766a40a8fe42e3ba4cd565c234b9a3"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "efdeb3ba0b2b035861e6acf82614dee0"
server_key_iv: "1c0b9ce9311e1794486de772d44de4ec"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.570812 25415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:52.571466 25430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.571616 25415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:52.571707 25415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "db766a40a8fe42e3ba4cd565c234b9a3"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "efdeb3ba0b2b035861e6acf82614dee0"
server_key_iv: "1c0b9ce9311e1794486de772d44de4ec"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.571766 25415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:52.579995 25415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:52.582031 25415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:52.582159 25415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:52.582358 25415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:52.582643 25415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:52.582675 25415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.582703 25415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:52.582727 25415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.588047 25415 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:43751
I20260619 02:49:52.588095 25543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:43751 every 8 connection(s)
I20260619 02:49:52.588387 25415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:52.589670 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 25415
I20260619 02:49:52.589771 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:52.589948 22163 external_mini_cluster.cc:1428] Setting key c5f49990210129724bcc86d20c3ef4ca
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
I20260619 02:49:52.597976 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.589020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:42751 (local address 127.21.164.254:44495)
0619 02:49:52.589061 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.589063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.589248 (+ 185us) server_negotiation.cc:407] Connection header received
0619 02:49:52.589762 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.589763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.589777 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.589822 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.590905 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.591402 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.592816 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.593070 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.593134 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.595326 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:52.595328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:52.595328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:52.595336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:52.596768 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.597117 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.597118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.597118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.597179 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.597374 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.597375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.597376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.597518 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:52.597587 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:52.597794 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 02:49:52.597879 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 02:49:52.598093 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.588988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:44495 (local address 127.21.164.194:42751)
0619 02:49:52.589125 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:52.589163 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:52.589668 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:52.589890 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:52.589894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:52.590621 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:52.590823 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.590829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.591485 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.591486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.592736 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.592739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.593144 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.593145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.593262 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.593727 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:52.593735 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:52.595240 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:52.596838 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:52.596843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:52.596845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:52.597049 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:52.597238 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:52.597239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:52.597239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:52.597322 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:52.597574 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:52.597578 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:52.597687 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 02:49:52.597761 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:49:52.597875 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260619 02:49:52.598620 25544 heartbeater.cc:347] Connected to a master server at 127.21.164.254:44495
I20260619 02:49:52.598702 25544 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:52.598861 25544 heartbeater.cc:511] Master 127.21.164.254:44495 requested a full tablet report, sending...
I20260619 02:49:52.599213 25219 ts_manager.cc:194] Registered new tserver with Master: db766a40a8fe42e3ba4cd565c234b9a3 (127.21.164.194:43751)
I20260619 02:49:52.599550 25219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:42751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:49:52.613648 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:44495
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:52.692137 25549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:52.692358 25549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:52.692390 25549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:52.693625 25549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:52.693677 25549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:52.693714 25549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:49:52.695173 25549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:44495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:52.695423 25549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:52.695634 25549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:52.700505 25555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.700500 25557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:52.700541 25554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:52.700591 25549 server_base.cc:1034] running on GCE node
I20260619 02:49:52.700954 25549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:52.701184 25549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:52.702320 25549 hybrid_clock.cc:648] HybridClock initialized: now 1781837392702305 us; error 37 us; skew 500 ppm
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:52.704859 25549 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:49:52.705276 25549 webserver.cc:460] Webserver started at http://127.21.164.195:44825/ using document root <none> and password file <none>
I20260619 02:49:52.705483 25549 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:52.705533 25549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:52.705653 25549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:52.706693 25549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "96123112140b49e0ac0dc98f9ede416b"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "63de9f27455fd3ce42ee2c6519e5313b"
server_key_iv: "89fb41fec028f644772cb165285c47aa"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.707020 25549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "96123112140b49e0ac0dc98f9ede416b"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "63de9f27455fd3ce42ee2c6519e5313b"
server_key_iv: "89fb41fec028f644772cb165285c47aa"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.708256 25549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:52.708962 25564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.709110 25549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.709189 25549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "96123112140b49e0ac0dc98f9ede416b"
format_stamp: "Formatted at 2026-06-19 02:49:52 on dist-test-slave-jt0t"
server_key: "63de9f27455fd3ce42ee2c6519e5313b"
server_key_iv: "89fb41fec028f644772cb165285c47aa"
server_key_version: "encryptionkey@0"
I20260619 02:49:52.709246 25549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:52.722785 25549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:52.724994 25549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:52.725134 25549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:52.725356 25549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:52.725661 25549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:52.725693 25549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.725721 25549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:52.725739 25549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:52.730983 25549 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:34361
I20260619 02:49:52.731043 25677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:34361 every 8 connection(s)
I20260619 02:49:52.731333 25549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:52.738184 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 25549
I20260619 02:49:52.738281 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:52.738448 22163 external_mini_cluster.cc:1428] Setting key 49f4b50d6f75f9e468c4064f33cf1b11
I20260619 02:49:52.739779 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.731997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:49001 (local address 127.21.164.254:44495)
0619 02:49:52.732058 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.732060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.732220 (+ 160us) server_negotiation.cc:407] Connection header received
0619 02:49:52.732709 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.732710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.732720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.732775 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.733680 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.734157 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.735073 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.735248 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.735290 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.737232 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:52.737233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:52.737234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:52.737240 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:52.738586 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.738933 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.738934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.738934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.738991 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.739199 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.739200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.739200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.739312 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:52.739364 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:52.739614 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 02:49:52.739693 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:49:52.739939 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.731956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:44495 (local address 127.21.164.195:49001)
0619 02:49:52.732104 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:52.732144 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:52.732624 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:52.732849 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:52.732854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:52.733408 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:52.733611 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.733616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.734219 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.734219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.735014 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:52.735016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.735300 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.735300 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:52.735395 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.735872 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:52.735879 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:52.737156 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:52.738650 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:52.738657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:52.738659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:52.738859 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:52.739054 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:52.739054 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:52.739055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:52.739144 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:52.739363 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:52.739368 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:52.739499 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 02:49:52.739585 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:49:52.739729 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260619 02:49:52.740334 25678 heartbeater.cc:347] Connected to a master server at 127.21.164.254:44495
I20260619 02:49:52.740412 25678 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:52.740578 25678 heartbeater.cc:511] Master 127.21.164.254:44495 requested a full tablet report, sending...
I20260619 02:49:52.740944 25219 ts_manager.cc:194] Registered new tserver with Master: 96123112140b49e0ac0dc98f9ede416b (127.21.164.195:34361)
I20260619 02:49:52.741274 25219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:49001
I20260619 02:49:52.749996 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837392, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:52.770958 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:52.764532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.21.164.254:44495)
0619 02:49:52.764600 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:52.764602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:52.764685 (+ 83us) server_negotiation.cc:407] Connection header received
0619 02:49:52.764794 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:52.764794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:52.764806 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:52.764864 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:52.765412 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.765907 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.766631 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:52.766818 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:52.766879 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:52.768961 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:52.768963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:52.768963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:52.768970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:52.770014 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.770284 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.770285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.770285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.770330 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:52.770449 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:52.770449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:52.770450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:52.770594 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:52.770672 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:52.770793 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 02:49:52.770866 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 02:49:52.771191 25219 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46964
I20260619 02:49:52.771601 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 25281
I20260619 02:49:52.775635 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 25415
I20260619 02:49:52.779001 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 25549
I20260619 02:49:52.782399 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 25187
2026-06-19T02:49:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1720 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25690](info): set up 2 sockets
Jun 19 02:49:52 dist-test-slave-jt0t krb5kdc[25690](info): commencing operation
krb5kdc: starting...
W20260619 02:49:53.728343 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:49:53 dist-test-slave-jt0t krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:49:53.839308 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:39033
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:39033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:53.918967 25703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:53.919173 25703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:53.919194 25703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:53.920790 25703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:53.920841 25703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:53.920854 25703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:53.920866 25703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:53.920890 25703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:53.922465 25703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:39033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:39033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:53.922724 25703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:53.922964 25703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:53.927636 25708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:53.927618 25711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:53.927647 25709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:53.927906 25703 server_base.cc:1034] running on GCE node
I20260619 02:49:53.928124 25703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:53.928383 25703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:53.929527 25703 hybrid_clock.cc:648] HybridClock initialized: now 1781837393929514 us; error 27 us; skew 500 ppm
Jun 19 02:49:53 dist-test-slave-jt0t krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837393, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:53.932139 25703 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:49:53.932597 25703 webserver.cc:460] Webserver started at http://127.21.164.254:39591/ using document root <none> and password file <none>
I20260619 02:49:53.932840 25703 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:53.932890 25703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:53.933020 25703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:53.934141 25703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "02dabfea68ce4ea5a5fa0e4d546df778"
format_stamp: "Formatted at 2026-06-19 02:49:53 on dist-test-slave-jt0t"
server_key: "1cc2c07340682ba36c92a5e4c3b97fb9"
server_key_iv: "94c5e6d7d20210ec72abe0b96e61783c"
server_key_version: "encryptionkey@0"
I20260619 02:49:53.934480 25703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "02dabfea68ce4ea5a5fa0e4d546df778"
format_stamp: "Formatted at 2026-06-19 02:49:53 on dist-test-slave-jt0t"
server_key: "1cc2c07340682ba36c92a5e4c3b97fb9"
server_key_iv: "94c5e6d7d20210ec72abe0b96e61783c"
server_key_version: "encryptionkey@0"
I20260619 02:49:53.935765 25703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:53.936457 25718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:53.936666 25703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:53.936758 25703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "02dabfea68ce4ea5a5fa0e4d546df778"
format_stamp: "Formatted at 2026-06-19 02:49:53 on dist-test-slave-jt0t"
server_key: "1cc2c07340682ba36c92a5e4c3b97fb9"
server_key_iv: "94c5e6d7d20210ec72abe0b96e61783c"
server_key_version: "encryptionkey@0"
I20260619 02:49:53.936823 25703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:53.943833 25703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:53.945935 25703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:53.946070 25703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:53.949707 25703 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:39033
I20260619 02:49:53.949750 25770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:39033 every 8 connection(s)
I20260619 02:49:53.950208 25703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:53.950855 25771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:53.953169 25771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778: Bootstrap starting.
I20260619 02:49:53.953744 25771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:53.953787 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 25703
I20260619 02:49:53.953889 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:53.954028 25771 log.cc:824] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:53.954059 22163 external_mini_cluster.cc:1428] Setting key 36e8ea596a42018946b88fcee9935593
I20260619 02:49:53.954738 25771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778: No bootstrap required, opened a new log
I20260619 02:49:53.956827 25771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dabfea68ce4ea5a5fa0e4d546df778" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39033 } }
I20260619 02:49:53.956962 25771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:53.956993 25771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02dabfea68ce4ea5a5fa0e4d546df778, State: Initialized, Role: FOLLOWER
I20260619 02:49:53.957118 25771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dabfea68ce4ea5a5fa0e4d546df778" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39033 } }
I20260619 02:49:53.957185 25771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:53.957216 25771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:53.957260 25771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:53.957764 25771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dabfea68ce4ea5a5fa0e4d546df778" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39033 } }
I20260619 02:49:53.957855 25771 leader_election.cc:302] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02dabfea68ce4ea5a5fa0e4d546df778; no voters:
I20260619 02:49:53.958006 25771 leader_election.cc:288] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:53.958050 25776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:53.958231 25776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [term 1 LEADER]: Becoming Leader. State: Replica: 02dabfea68ce4ea5a5fa0e4d546df778, State: Running, Role: LEADER
I20260619 02:49:53.958369 25776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dabfea68ce4ea5a5fa0e4d546df778" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39033 } }
I20260619 02:49:53.958712 25776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02dabfea68ce4ea5a5fa0e4d546df778. Latest consensus state: current_term: 1 leader_uuid: "02dabfea68ce4ea5a5fa0e4d546df778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dabfea68ce4ea5a5fa0e4d546df778" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39033 } } }
I20260619 02:49:53.958741 25778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02dabfea68ce4ea5a5fa0e4d546df778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dabfea68ce4ea5a5fa0e4d546df778" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39033 } } }
I20260619 02:49:53.958252 25771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:53.958839 25776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:53.958932 25778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778 [sys.catalog]: This master's current role is: LEADER
Jun 19 02:49:53 dist-test-slave-jt0t krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
W20260619 02:49:53.961081 25791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:49:53.961143 25791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:49:53.961184 25792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:53.961364 25792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:53.963124 25792 catalog_manager.cc:1347] Generated new cluster ID: 99d50e64aabc4d97b47f4eaf2de58abd
I20260619 02:49:53.963172 25792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:53.964448 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:53.954531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.21.164.254:39033)
0619 02:49:53.954689 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:53.954700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:53.954726 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:49:53.955168 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:53.955175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:53.955233 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:53.955354 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:53.955972 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.956889 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.957701 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:53.958240 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:53.958653 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:53.961097 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:53.961102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:53.961102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:53.961114 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:53.962838 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:53.963241 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:53.963244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:53.963244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:53.963346 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:53.963562 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:53.963562 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:53.963563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:53.963862 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:53.964022 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:53.964092 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 02:49:53.964225 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260619 02:49:53.990633 25792 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:53.991405 25792 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:53.995023 25792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02dabfea68ce4ea5a5fa0e4d546df778: Generated new TSK 0
I20260619 02:49:53.995211 25792 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:49:54.027801 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39033
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:54.108732 25797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:54.108955 25797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:54.108979 25797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:54.110146 25797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:54.110189 25797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:54.110219 25797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:49:54.111750 25797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:54.111995 25797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:54.112212 25797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:54.116969 25805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:54.117122 25797 server_base.cc:1034] running on GCE node
W20260619 02:49:54.117029 25803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:54.117048 25802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:54.117488 25797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:54.117727 25797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:54.118860 25797 hybrid_clock.cc:648] HybridClock initialized: now 1781837394118852 us; error 30 us; skew 500 ppm
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837394, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:54.121443 25797 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:49:54.121886 25797 webserver.cc:460] Webserver started at http://127.21.164.193:40635/ using document root <none> and password file <none>
I20260619 02:49:54.122118 25797 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:54.122164 25797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:54.122287 25797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:54.123427 25797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "1799591cd8244452b4680c2e016755e7"
format_stamp: "Formatted at 2026-06-19 02:49:54 on dist-test-slave-jt0t"
server_key: "93f2f3c1e95199bb56d25e597b36a344"
server_key_iv: "adfb3e628c244d64ecf3ee1cdc51f349"
server_key_version: "encryptionkey@0"
I20260619 02:49:54.123766 25797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "1799591cd8244452b4680c2e016755e7"
format_stamp: "Formatted at 2026-06-19 02:49:54 on dist-test-slave-jt0t"
server_key: "93f2f3c1e95199bb56d25e597b36a344"
server_key_iv: "adfb3e628c244d64ecf3ee1cdc51f349"
server_key_version: "encryptionkey@0"
I20260619 02:49:54.125066 25797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:54.125798 25812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:54.125946 25797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:54.126015 25797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "1799591cd8244452b4680c2e016755e7"
format_stamp: "Formatted at 2026-06-19 02:49:54 on dist-test-slave-jt0t"
server_key: "93f2f3c1e95199bb56d25e597b36a344"
server_key_iv: "adfb3e628c244d64ecf3ee1cdc51f349"
server_key_version: "encryptionkey@0"
I20260619 02:49:54.126080 25797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:54.137425 25797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:54.139606 25797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:54.139731 25797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:54.139947 25797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:54.140233 25797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:54.140265 25797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:54.140288 25797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:54.140311 25797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:54.145763 25797 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:38431
I20260619 02:49:54.145809 25925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:38431 every 8 connection(s)
I20260619 02:49:54.146111 25797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837394, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:54.152570 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 25797
I20260619 02:49:54.152710 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:54.152904 22163 external_mini_cluster.cc:1428] Setting key b9d8d9ebc37bb3917cf87473511c896e
I20260619 02:49:54.154886 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:54.146776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:50587 (local address 127.21.164.254:39033)
0619 02:49:54.146848 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:54.146850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:54.147015 (+ 165us) server_negotiation.cc:407] Connection header received
0619 02:49:54.147964 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:54.147965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:54.147975 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:54.148026 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:54.148592 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.149107 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.150031 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.150220 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.150263 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.152220 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:54.152222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:54.152222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:54.152229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:54.153640 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:54.154011 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:54.154012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:54.154012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:54.154075 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:54.154268 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:54.154268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:54.154269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:54.154394 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:54.154448 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:54.154690 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 02:49:54.154793 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:49:54.155035 25928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:54.146748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39033 (local address 127.21.164.193:50587)
0619 02:49:54.146888 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:54.146936 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:54.147866 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:54.148084 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:54.148087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:54.148336 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:54.148528 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.148532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.149164 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.149164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.149971 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.149973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.150273 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.150274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.150368 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.150805 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:54.150813 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:54.152128 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:54.153711 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:54.153718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:54.153720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:54.153947 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:54.154131 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:54.154131 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:54.154132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:54.154222 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:54.154448 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:54.154452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:54.154565 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 02:49:54.154650 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 02:49:54.154802 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260619 02:49:54.155457 25926 heartbeater.cc:347] Connected to a master server at 127.21.164.254:39033
I20260619 02:49:54.155570 25926 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:54.155761 25926 heartbeater.cc:511] Master 127.21.164.254:39033 requested a full tablet report, sending...
I20260619 02:49:54.156250 25735 ts_manager.cc:194] Registered new tserver with Master: 1799591cd8244452b4680c2e016755e7 (127.21.164.193:38431)
I20260619 02:49:54.156911 25735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:50587
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:49:54.176267 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39033
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:54.256805 25931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:54.257017 25931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:54.257041 25931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:54.258165 25931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:54.258200 25931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:54.258224 25931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:49:54.259752 25931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:54.259967 25931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:54.260144 25931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:54.265012 25937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:54.265084 25936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:54.265065 25931 server_base.cc:1034] running on GCE node
W20260619 02:49:54.265023 25939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:54.265456 25931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:54.265717 25931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:54.266860 25931 hybrid_clock.cc:648] HybridClock initialized: now 1781837394266845 us; error 26 us; skew 500 ppm
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837394, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:54.273952 25931 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:49:54.274390 25931 webserver.cc:460] Webserver started at http://127.21.164.194:45895/ using document root <none> and password file <none>
I20260619 02:49:54.274575 25931 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:54.274614 25931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:54.274715 25931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:54.275810 25931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "9e1dcb4c0c2746ea91a708b7a046b467"
format_stamp: "Formatted at 2026-06-19 02:49:54 on dist-test-slave-jt0t"
server_key: "9971e551cd6ddfead2d9a3d385b22537"
server_key_iv: "cb04487fc4b7ebfe12cddc49b66e5bab"
server_key_version: "encryptionkey@0"
I20260619 02:49:54.276124 25931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "9e1dcb4c0c2746ea91a708b7a046b467"
format_stamp: "Formatted at 2026-06-19 02:49:54 on dist-test-slave-jt0t"
server_key: "9971e551cd6ddfead2d9a3d385b22537"
server_key_iv: "cb04487fc4b7ebfe12cddc49b66e5bab"
server_key_version: "encryptionkey@0"
I20260619 02:49:54.277562 25931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:54.278311 25946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:54.278465 25931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:54.278537 25931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "9e1dcb4c0c2746ea91a708b7a046b467"
format_stamp: "Formatted at 2026-06-19 02:49:54 on dist-test-slave-jt0t"
server_key: "9971e551cd6ddfead2d9a3d385b22537"
server_key_iv: "cb04487fc4b7ebfe12cddc49b66e5bab"
server_key_version: "encryptionkey@0"
I20260619 02:49:54.278592 25931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:54.295825 25931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:54.297791 25931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:54.297901 25931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:54.298097 25931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:54.298343 25931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:54.298370 25931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:54.298390 25931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:54.298405 25931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:54.303655 25931 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:35851
I20260619 02:49:54.303699 26059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:35851 every 8 connection(s)
I20260619 02:49:54.303995 25931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837394, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:54.311156 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 25931
I20260619 02:49:54.311285 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:54.311475 22163 external_mini_cluster.cc:1428] Setting key b35bcf7be747f5c0f8f389f9af980f1d
I20260619 02:49:54.312919 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:54.304641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:58743 (local address 127.21.164.254:39033)
0619 02:49:54.304711 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:54.304713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:54.304915 (+ 202us) server_negotiation.cc:407] Connection header received
0619 02:49:54.305773 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:54.305775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:54.305786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:54.305836 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:54.306409 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.306884 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.307781 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.307964 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.308009 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.310002 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:54.310003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:54.310004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:54.310010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:54.311467 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:54.311923 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:54.311924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:54.311925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:54.311987 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:54.312259 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:54.312260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:54.312260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:54.312406 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:54.312471 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:54.312741 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 02:49:54.312827 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:49:54.313063 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:54.304625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39033 (local address 127.21.164.194:58743)
0619 02:49:54.304792 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:54.304828 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:54.305692 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:54.305891 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:54.305894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:54.306162 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:54.306348 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.306353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.306952 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.306953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.307720 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.307722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.308019 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.308020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.308112 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.308590 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:54.308598 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:54.309889 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:54.311548 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:54.311555 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:54.311558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:54.311840 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:54.312054 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:54.312054 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:54.312055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:54.312190 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:54.312470 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:54.312474 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:54.312591 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 02:49:54.312719 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 02:49:54.312852 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260619 02:49:54.313508 26060 heartbeater.cc:347] Connected to a master server at 127.21.164.254:39033
I20260619 02:49:54.313688 26060 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:54.313874 26060 heartbeater.cc:511] Master 127.21.164.254:39033 requested a full tablet report, sending...
I20260619 02:49:54.314235 25735 ts_manager.cc:194] Registered new tserver with Master: 9e1dcb4c0c2746ea91a708b7a046b467 (127.21.164.194:35851)
I20260619 02:49:54.314723 25735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:58743
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:49:54.337363 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39033
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:54.416288 26065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:54.416496 26065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:54.416519 26065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:54.417711 26065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:54.417757 26065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:54.417789 26065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:49:54.419283 26065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:54.419519 26065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:54.419737 26065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:54.424507 26073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:54.424558 26071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:54.424502 26070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:54.424748 26065 server_base.cc:1034] running on GCE node
I20260619 02:49:54.424952 26065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:54.425180 26065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:54.426316 26065 hybrid_clock.cc:648] HybridClock initialized: now 1781837394426304 us; error 37 us; skew 500 ppm
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837394, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:54.428687 26065 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:49:54.429150 26065 webserver.cc:460] Webserver started at http://127.21.164.195:42057/ using document root <none> and password file <none>
I20260619 02:49:54.429365 26065 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:54.429421 26065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:54.429548 26065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:54.430611 26065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "62fef45f7a9e4d648dbd3e82f18003b3"
format_stamp: "Formatted at 2026-06-19 02:49:54 on dist-test-slave-jt0t"
server_key: "94e7189f3b12f6463b998809e02d905e"
server_key_iv: "1a95dc2500efd74d9324e98c5c929202"
server_key_version: "encryptionkey@0"
I20260619 02:49:54.430940 26065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "62fef45f7a9e4d648dbd3e82f18003b3"
format_stamp: "Formatted at 2026-06-19 02:49:54 on dist-test-slave-jt0t"
server_key: "94e7189f3b12f6463b998809e02d905e"
server_key_iv: "1a95dc2500efd74d9324e98c5c929202"
server_key_version: "encryptionkey@0"
I20260619 02:49:54.432194 26065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:54.432917 26080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:54.433063 26065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:54.433142 26065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "62fef45f7a9e4d648dbd3e82f18003b3"
format_stamp: "Formatted at 2026-06-19 02:49:54 on dist-test-slave-jt0t"
server_key: "94e7189f3b12f6463b998809e02d905e"
server_key_iv: "1a95dc2500efd74d9324e98c5c929202"
server_key_version: "encryptionkey@0"
I20260619 02:49:54.433202 26065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:54.442597 26065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:54.444571 26065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:54.444732 26065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:54.444973 26065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:54.445272 26065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:54.445305 26065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:54.445333 26065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:54.445358 26065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:54.450779 26065 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:45765
I20260619 02:49:54.450862 26193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:45765 every 8 connection(s)
I20260619 02:49:54.451148 26065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:54.451726 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 26065
I20260619 02:49:54.451821 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:54.452026 22163 external_mini_cluster.cc:1428] Setting key becd32b51138dc6c11b3a223ca07ba74
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837394, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:54.460031 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:54.451871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:60633 (local address 127.21.164.254:39033)
0619 02:49:54.451930 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:54.451932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:54.452154 (+ 222us) server_negotiation.cc:407] Connection header received
0619 02:49:54.453240 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:54.453241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:54.453251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:54.453290 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:54.453887 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.454384 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.455342 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.455529 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.455578 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.457561 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:54.457562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:54.457562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:54.457570 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:54.458753 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:54.459123 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:54.459123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:54.459124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:54.459172 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:54.459357 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:54.459358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:54.459358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:54.459471 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:54.459521 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:54.459851 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 02:49:54.459937 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:49:54.460222 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:54.451842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39033 (local address 127.21.164.195:60633)
0619 02:49:54.452018 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:54.452070 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:54.453153 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:54.453352 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:54.453355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:54.453612 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:54.453817 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.453822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.454453 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.454454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.455273 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:54.455276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.455587 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.455588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:54.455684 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.456168 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:54.456177 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:54.457472 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:54.458816 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:54.458822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:54.458824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:54.459074 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:54.459217 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:54.459217 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:54.459218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:54.459319 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:54.459531 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:54.459538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:54.459721 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 02:49:54.459814 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:49:54.459973 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260619 02:49:54.460696 26194 heartbeater.cc:347] Connected to a master server at 127.21.164.254:39033
I20260619 02:49:54.460794 26194 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:54.460971 26194 heartbeater.cc:511] Master 127.21.164.254:39033 requested a full tablet report, sending...
I20260619 02:49:54.461305 25735 ts_manager.cc:194] Registered new tserver with Master: 62fef45f7a9e4d648dbd3e82f18003b3 (127.21.164.195:45765)
I20260619 02:49:54.461637 25735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:60633
I20260619 02:49:54.463580 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:49:54.485281 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:54.477767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.21.164.193:38431)
0619 02:49:54.477922 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:54.477928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:54.477941 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:49:54.477966 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:54.477968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:54.477981 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:54.478054 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:54.478556 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.479283 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.480018 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.480386 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.480452 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.482776 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:54.482789 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:54.482789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:54.482796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:54.484350 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:54.484625 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:54.484628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:54.484628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:54.484710 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:54.484860 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:54.484861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:54.484861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:54.484996 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:54.485103 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:54.485136 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 02:49:54.485196 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:54.492591 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:54.486541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.21.164.254:39033)
0619 02:49:54.486592 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:54.486594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:54.486601 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:49:54.486652 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:54.486653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:54.486662 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:54.486702 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:54.487098 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.487578 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.488256 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:54.488449 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:54.488506 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:54.490571 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:54.490573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:54.490574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:54.490580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:54.491634 (+ 1054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:54.491936 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:54.491937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:54.491937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:54.491978 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:54.492179 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:54.492179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:54.492179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:54.492309 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:54.492379 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:54.492463 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 02:49:54.492520 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 02:49:54.492880 25735 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52920
I20260619 02:49:54.493227 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 25797
I20260619 02:49:54.497430 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 25931
I20260619 02:49:54.500823 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 26065
I20260619 02:49:54.504354 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 25703
2026-06-19T02:49:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1723 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[26206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[26206](info): set up 2 sockets
Jun 19 02:49:54 dist-test-slave-jt0t krb5kdc[26206](info): commencing operation
krb5kdc: starting...
W20260619 02:49:55.448351 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:49:55 dist-test-slave-jt0t krb5kdc[26206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:49:55.561764 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:37543
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:37543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:55.641546 26219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:55.641768 26219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:55.641791 26219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:55.642951 26219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:55.642987 26219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:55.642999 26219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:55.643010 26219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:55.643020 26219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:55.644487 26219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:37543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:37543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:55.644733 26219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:55.644924 26219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:55.649564 26224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:55.649588 26219 server_base.cc:1034] running on GCE node
W20260619 02:49:55.649583 26225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:55.649571 26227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:55.649959 26219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:55.650257 26219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:55.651436 26219 hybrid_clock.cc:648] HybridClock initialized: now 1781837395651417 us; error 31 us; skew 500 ppm
Jun 19 02:49:55 dist-test-slave-jt0t krb5kdc[26206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837395, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:55.653959 26219 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:49:55.654286 26219 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:55.654338 26219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:55.654467 26219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:55.655530 26219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "3502f954b024448f9301863e2adf469e"
format_stamp: "Formatted at 2026-06-19 02:49:55 on dist-test-slave-jt0t"
server_key: "ee958cf9e934bcf83cf730df1b4be807"
server_key_iv: "208f5b02fb8ebae05114dc1e3ba0811e"
server_key_version: "encryptionkey@0"
I20260619 02:49:55.655876 26219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "3502f954b024448f9301863e2adf469e"
format_stamp: "Formatted at 2026-06-19 02:49:55 on dist-test-slave-jt0t"
server_key: "ee958cf9e934bcf83cf730df1b4be807"
server_key_iv: "208f5b02fb8ebae05114dc1e3ba0811e"
server_key_version: "encryptionkey@0"
I20260619 02:49:55.657140 26219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 02:49:55.657794 26233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:55.657951 26219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:55.658030 26219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "3502f954b024448f9301863e2adf469e"
format_stamp: "Formatted at 2026-06-19 02:49:55 on dist-test-slave-jt0t"
server_key: "ee958cf9e934bcf83cf730df1b4be807"
server_key_iv: "208f5b02fb8ebae05114dc1e3ba0811e"
server_key_version: "encryptionkey@0"
I20260619 02:49:55.658089 26219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:55.666318 26219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:55.668320 26219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:55.668457 26219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:55.672120 26219 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:37543
I20260619 02:49:55.672143 26285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:37543 every 8 connection(s)
I20260619 02:49:55.672406 26219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:55.673316 26286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:55.675595 26286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e: Bootstrap starting.
I20260619 02:49:55.676151 26286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:55.676175 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 26219
I20260619 02:49:55.676275 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:55.676440 22163 external_mini_cluster.cc:1428] Setting key c4bfa6d3c31e96d216dd1af53161c22d
I20260619 02:49:55.676435 26286 log.cc:824] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:55.677161 26286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e: No bootstrap required, opened a new log
I20260619 02:49:55.679204 26286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3502f954b024448f9301863e2adf469e" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37543 } }
I20260619 02:49:55.679411 26286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:55.679471 26286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3502f954b024448f9301863e2adf469e, State: Initialized, Role: FOLLOWER
I20260619 02:49:55.679611 26286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3502f954b024448f9301863e2adf469e" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37543 } }
I20260619 02:49:55.679683 26286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:55.679704 26286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:55.679738 26286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:55.680284 26286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3502f954b024448f9301863e2adf469e" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37543 } }
I20260619 02:49:55.680377 26286 leader_election.cc:302] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3502f954b024448f9301863e2adf469e; no voters:
I20260619 02:49:55.680526 26286 leader_election.cc:288] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:55.680565 26291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:55.680724 26291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [term 1 LEADER]: Becoming Leader. State: Replica: 3502f954b024448f9301863e2adf469e, State: Running, Role: LEADER
I20260619 02:49:55.680827 26291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3502f954b024448f9301863e2adf469e" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37543 } }
I20260619 02:49:55.680825 26286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:55.681766 26293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3502f954b024448f9301863e2adf469e. Latest consensus state: current_term: 1 leader_uuid: "3502f954b024448f9301863e2adf469e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3502f954b024448f9301863e2adf469e" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37543 } } }
I20260619 02:49:55.681850 26293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:55.682060 26303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:55.682154 26292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3502f954b024448f9301863e2adf469e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3502f954b024448f9301863e2adf469e" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37543 } } }
I20260619 02:49:55.682405 26292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:55.682679 26303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 02:49:55 dist-test-slave-jt0t krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:55.684770 26303 catalog_manager.cc:1347] Generated new cluster ID: 5e21fd46a5114bada17bb43c7935fc32
I20260619 02:49:55.684824 26303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:55.686674 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:55.676960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.21.164.254:37543)
0619 02:49:55.677132 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:55.677142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:55.677167 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:49:55.677619 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:55.677626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:55.677683 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:55.677805 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:55.678412 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.679292 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.680150 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.680621 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.680783 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:55.683601 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:55.683605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:55.683605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:55.683617 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:55.685291 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:55.685608 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:55.685611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:55.685612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:55.685695 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:55.685861 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:55.685862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:55.685862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:55.686097 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:55.686244 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:55.686335 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 02:49:55.686465 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260619 02:49:55.733912 26303 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:55.734483 26303 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:55.739440 26303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e: Generated new TSK 0
I20260619 02:49:55.739619 26303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:49:55.773397 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37543
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:55.853497 26312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:55.853710 26312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:55.853734 26312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:55.854908 26312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:55.854952 26312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:55.854979 26312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:49:55.856493 26312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:55.856751 26312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:55.856974 26312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:55.861699 26320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:55.861744 26318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:55.861733 26317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:55.862016 26312 server_base.cc:1034] running on GCE node
I20260619 02:49:55.862229 26312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:55.862452 26312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:55.863587 26312 hybrid_clock.cc:648] HybridClock initialized: now 1781837395863573 us; error 28 us; skew 500 ppm
Jun 19 02:49:55 dist-test-slave-jt0t krb5kdc[26206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837395, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:55.866128 26312 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:49:55.866444 26312 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:55.866495 26312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:55.866614 26312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:55.867631 26312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "1cbe5133f9ef4cec88004434e981af05"
format_stamp: "Formatted at 2026-06-19 02:49:55 on dist-test-slave-jt0t"
server_key: "049bbaf033bc2b4fe589d048e4110e05"
server_key_iv: "e898fb6f804dd14433063b70e57c7813"
server_key_version: "encryptionkey@0"
I20260619 02:49:55.867962 26312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "1cbe5133f9ef4cec88004434e981af05"
format_stamp: "Formatted at 2026-06-19 02:49:55 on dist-test-slave-jt0t"
server_key: "049bbaf033bc2b4fe589d048e4110e05"
server_key_iv: "e898fb6f804dd14433063b70e57c7813"
server_key_version: "encryptionkey@0"
I20260619 02:49:55.869199 26312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:55.869975 26326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:55.870164 26312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:55.870254 26312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "1cbe5133f9ef4cec88004434e981af05"
format_stamp: "Formatted at 2026-06-19 02:49:55 on dist-test-slave-jt0t"
server_key: "049bbaf033bc2b4fe589d048e4110e05"
server_key_iv: "e898fb6f804dd14433063b70e57c7813"
server_key_version: "encryptionkey@0"
I20260619 02:49:55.870317 26312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:55.884037 26312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:55.886245 26312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:55.886382 26312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:55.886600 26312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:55.886905 26312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:55.886940 26312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:55.886970 26312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:55.886993 26312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:55.892309 26312 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:45361
I20260619 02:49:55.892374 26439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:45361 every 8 connection(s)
I20260619 02:49:55.892714 26312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:55.898139 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 26312
I20260619 02:49:55.898265 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:55.898437 22163 external_mini_cluster.cc:1428] Setting key 2eb190da19960165cfa3fa62ce3b242f
Jun 19 02:49:55 dist-test-slave-jt0t krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837395, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:55.902019 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:55.893451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:53885 (local address 127.21.164.254:37543)
0619 02:49:55.893511 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:55.893513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:55.893694 (+ 181us) server_negotiation.cc:407] Connection header received
0619 02:49:55.894161 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:55.894162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:55.894171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:55.894228 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:55.895161 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.895695 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.896707 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.896915 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.896965 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:55.899163 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:55.899164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:55.899165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:55.899172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:55.900603 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:55.901116 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:55.901116 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:55.901117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:55.901165 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:55.901419 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:55.901420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:55.901420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:55.901530 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:55.901573 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:55.901889 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 02:49:55.901955 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 02:49:55.902242 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:55.893415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:37543 (local address 127.21.164.193:53885)
0619 02:49:55.893574 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:55.893616 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:55.894082 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:55.894294 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:55.894297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:55.894561 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:55.895062 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:55.895067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.895797 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.895798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:55.896631 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:55.896633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:55.897005 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:55.897007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:55.897118 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:55.897592 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:55.897600 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:55.899085 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:55.900708 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:55.900716 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:55.900718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:55.901043 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:55.901220 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:55.901221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:55.901221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:55.901355 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:55.901582 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:55.901587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:55.901747 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 02:49:55.901870 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 02:49:55.902027 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260619 02:49:55.902720 26440 heartbeater.cc:347] Connected to a master server at 127.21.164.254:37543
I20260619 02:49:55.902832 26440 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:55.902988 26440 heartbeater.cc:511] Master 127.21.164.254:37543 requested a full tablet report, sending...
I20260619 02:49:55.903492 26250 ts_manager.cc:194] Registered new tserver with Master: 1cbe5133f9ef4cec88004434e981af05 (127.21.164.193:45361)
I20260619 02:49:55.904078 26250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:53885
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:49:55.923034 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37543
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:56.002317 26445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:56.002532 26445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:56.002555 26445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:56.003737 26445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:56.003785 26445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:56.003813 26445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:49:56.005357 26445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:56.005597 26445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:56.005810 26445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:56.010716 26453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:56.010795 26450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:56.010716 26451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:56.010876 26445 server_base.cc:1034] running on GCE node
I20260619 02:49:56.011164 26445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:56.011425 26445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:56.012593 26445 hybrid_clock.cc:648] HybridClock initialized: now 1781837396012577 us; error 24 us; skew 500 ppm
Jun 19 02:49:56 dist-test-slave-jt0t krb5kdc[26206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837396, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:56.015043 26445 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:49:56.015409 26445 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:56.015463 26445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:56.015594 26445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:56.016640 26445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "986bf36105d3430a95595e8b4d34a3c3"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "0031760038489f66e60ad7898b6d1921"
server_key_iv: "d45c4801a7ff1c2f84d4afdbe3428e44"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.017038 26445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "986bf36105d3430a95595e8b4d34a3c3"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "0031760038489f66e60ad7898b6d1921"
server_key_iv: "d45c4801a7ff1c2f84d4afdbe3428e44"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.018306 26445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:56.018997 26459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.019155 26445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:56.019236 26445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "986bf36105d3430a95595e8b4d34a3c3"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "0031760038489f66e60ad7898b6d1921"
server_key_iv: "d45c4801a7ff1c2f84d4afdbe3428e44"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.019308 26445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:56.035336 26445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:56.037624 26445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:56.037770 26445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:56.037997 26445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:56.038317 26445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:56.038353 26445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.038384 26445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:56.038405 26445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.043720 26445 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:35597
I20260619 02:49:56.043769 26572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:35597 every 8 connection(s)
I20260619 02:49:56.044036 26445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:56.047927 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 26445
I20260619 02:49:56.048044 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:56.048223 22163 external_mini_cluster.cc:1428] Setting key 2a1b5c2a1262b54ccc20fda3a147330b
Jun 19 02:49:56 dist-test-slave-jt0t krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837396, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:56.053707 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.044739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:48921 (local address 127.21.164.254:37543)
0619 02:49:56.044800 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.044802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.045015 (+ 213us) server_negotiation.cc:407] Connection header received
0619 02:49:56.045536 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.045537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.045549 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.045608 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.046574 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.047077 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.048091 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.048347 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.048411 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.050758 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:56.050760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:56.050761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:56.050770 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:56.052246 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:56.052742 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:56.052743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:56.052744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:56.052808 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:56.053051 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:56.053052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:56.053053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:56.053171 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:56.053227 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.053507 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.053601 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:49:56.053874 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.044732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:37543 (local address 127.21.164.194:48921)
0619 02:49:56.044896 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.044934 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.045438 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.045683 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.045688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.045957 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:56.046481 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.046488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.047158 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.047160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.048011 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.048014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.048451 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.048453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.048586 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.049157 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:56.049167 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:56.050660 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:56.052332 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:56.052338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:56.052340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:56.052630 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:56.052868 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:56.052869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:56.052869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:56.052978 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:56.053226 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:56.053232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:56.053385 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.053468 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.053633 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260619 02:49:56.054471 26573 heartbeater.cc:347] Connected to a master server at 127.21.164.254:37543
I20260619 02:49:56.054587 26573 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:56.054757 26573 heartbeater.cc:511] Master 127.21.164.254:37543 requested a full tablet report, sending...
I20260619 02:49:56.055115 26250 ts_manager.cc:194] Registered new tserver with Master: 986bf36105d3430a95595e8b4d34a3c3 (127.21.164.194:35597)
I20260619 02:49:56.055477 26250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:48921
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:49:56.073592 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37543
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:56.153299 26578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:56.153501 26578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:56.153522 26578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:56.154680 26578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:56.154721 26578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:56.154748 26578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:49:56.156205 26578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:56.156432 26578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:56.156672 26578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:56.161562 26584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:56.161605 26586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:56.161600 26578 server_base.cc:1034] running on GCE node
W20260619 02:49:56.161567 26583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:56.161954 26578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:56.162194 26578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:56.163342 26578 hybrid_clock.cc:648] HybridClock initialized: now 1781837396163327 us; error 28 us; skew 500 ppm
Jun 19 02:49:56 dist-test-slave-jt0t krb5kdc[26206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837396, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:56.165848 26578 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:49:56.166180 26578 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:56.166234 26578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:56.166384 26578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:56.167430 26578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "097d6337c7744808a706996c3b9c166b"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "72bf5c14f49e6db7404d79adc9b1ee20"
server_key_iv: "ce7887a55e3675877c734b6a9b36cae3"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.167761 26578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "097d6337c7744808a706996c3b9c166b"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "72bf5c14f49e6db7404d79adc9b1ee20"
server_key_iv: "ce7887a55e3675877c734b6a9b36cae3"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.168951 26578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:49:56.169616 26592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.169759 26578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.169845 26578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "097d6337c7744808a706996c3b9c166b"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "72bf5c14f49e6db7404d79adc9b1ee20"
server_key_iv: "ce7887a55e3675877c734b6a9b36cae3"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.169901 26578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:56.179596 26578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:56.181664 26578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:56.181929 26578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:56.182154 26578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:56.182497 26578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:56.182538 26578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.182567 26578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:56.182590 26578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.187942 26578 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:37093
I20260619 02:49:56.188002 26705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:37093 every 8 connection(s)
I20260619 02:49:56.188241 26578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:49:56 dist-test-slave-jt0t krb5kdc[26206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837396, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:56.196889 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.188934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:60491 (local address 127.21.164.254:37543)
0619 02:49:56.188974 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.188976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.189176 (+ 200us) server_negotiation.cc:407] Connection header received
0619 02:49:56.189626 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.189627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.189638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.189687 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.190600 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.191093 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.192108 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.192329 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.192384 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.194497 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:56.194499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:56.194499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:56.194505 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:56.195670 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:56.196012 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:56.196013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:56.196013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:56.196067 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:56.196284 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:56.196285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:56.196286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:56.196427 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:56.196487 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.196747 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.196819 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 02:49:56.197026 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.188918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:37543 (local address 127.21.164.195:60491)
0619 02:49:56.189043 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.189079 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.189544 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.189743 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.189746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.190009 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:56.190507 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.190513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.191178 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.191179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.192025 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.192027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.192393 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.192394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.192490 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.193035 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:56.193046 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:56.194400 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:56.195735 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:56.195740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:56.195742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:56.195946 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:56.196123 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:56.196124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:56.196124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:56.196211 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:56.196483 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:56.196486 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:56.196598 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.196717 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.196842 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
I20260619 02:49:56.197490 26706 heartbeater.cc:347] Connected to a master server at 127.21.164.254:37543
I20260619 02:49:56.197575 26706 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:56.197688 26706 heartbeater.cc:511] Master 127.21.164.254:37543 requested a full tablet report, sending...
I20260619 02:49:56.198071 26250 ts_manager.cc:194] Registered new tserver with Master: 097d6337c7744808a706996c3b9c166b (127.21.164.195:37093)
I20260619 02:49:56.198412 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 26578
I20260619 02:49:56.198499 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:56.198524 26250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:60491
I20260619 02:49:56.198668 22163 external_mini_cluster.cc:1428] Setting key 5895763edeb4479d6a675387e39bc40a
I20260619 02:49:56.199440 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:56.205911 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.200613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33514 (local address 127.21.164.254:37543)
0619 02:49:56.200685 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.200687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.200725 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:49:56.200788 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.200789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.200798 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.200853 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.201353 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.201818 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.202509 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.202701 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.202751 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.203809 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:56.203810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:56.203811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:56.203817 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:56.204916 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:56.205206 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:56.205207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:56.205208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:56.205257 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:56.205420 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:56.205421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:56.205421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:56.205553 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:56.205639 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.205739 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.205813 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:49:56.207248 26250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:56.208848 26250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:56.214808 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.210727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:45361 (local address 127.0.0.1:33708)
0619 02:49:56.210846 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.210877 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.210940 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.211444 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.211445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.211447 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:56.211500 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.211501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.212448 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.212449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.213670 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.213673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.214542 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.214543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.214635 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.214648 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.214706 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.214746 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260619 02:49:56.214874 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.210916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.21.164.193:45361)
0619 02:49:56.211237 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.211242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.211253 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:56.211293 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.211297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.211313 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.211447 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:56.211552 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.212358 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.213748 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.214460 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.214537 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.214587 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.214735 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.214780 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":74,"threads_started":1}
I20260619 02:49:56.215199 26716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.210922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:35597 (local address 127.0.0.1:51506)
0619 02:49:56.211276 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.211285 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.211341 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.211585 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.211586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.211587 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:56.211623 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.211624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.212927 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.212928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.213833 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.213836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.214999 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.215000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.215051 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.215054 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.215101 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.215137 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":27,"thread_start_us":45,"threads_started":1}
I20260619 02:49:56.215256 26719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.211098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.21.164.194:35597)
0619 02:49:56.211271 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.211274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.211331 (+ 57us) server_negotiation.cc:407] Connection header received
0619 02:49:56.211417 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.211419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.211433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.211501 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:56.211692 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.212823 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.213929 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.214465 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.214532 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.214588 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.215131 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.215175 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260619 02:49:56.215517 26374 tablet_service.cc:1467] Processing CreateTablet for tablet e0cf9a9235244c7394d6d66c6906fd1c (DEFAULT_TABLE table=test-table [id=9ea1a136903e430b96dcd1f3011fc52d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:56.215832 26374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0cf9a9235244c7394d6d66c6906fd1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:56.217006 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.210727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:37093 (local address 127.0.0.1:35010)
0619 02:49:56.211032 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.211040 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.211069 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.211344 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.211347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.211349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:56.211421 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.211423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.212918 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.212919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.214010 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.214013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.214846 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.214846 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.214903 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.214907 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.216895 (+ 1988us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.216942 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":29,"thread_start_us":61,"threads_started":1}
I20260619 02:49:56.217192 26718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.211026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.21.164.195:37093)
0619 02:49:56.211164 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.211168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.211176 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:56.211210 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.211213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.211242 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.211599 (+ 357us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:56.211678 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.212805 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.214086 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.214761 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.214832 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.215167 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.216928 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.216972 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260619 02:49:56.217273 26507 tablet_service.cc:1467] Processing CreateTablet for tablet e0cf9a9235244c7394d6d66c6906fd1c (DEFAULT_TABLE table=test-table [id=9ea1a136903e430b96dcd1f3011fc52d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:56.217624 26507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0cf9a9235244c7394d6d66c6906fd1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:56.219005 26720 tablet_bootstrap.cc:492] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05: Bootstrap starting.
I20260619 02:49:56.219638 26720 tablet_bootstrap.cc:654] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:56.220216 26720 log.cc:824] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:56.220759 26722 tablet_bootstrap.cc:492] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: Bootstrap starting.
I20260619 02:49:56.221025 26720 tablet_bootstrap.cc:492] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05: No bootstrap required, opened a new log
I20260619 02:49:56.221100 26720 ts_tablet_manager.cc:1397] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:56.221393 26722 tablet_bootstrap.cc:654] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:56.221354 26640 tablet_service.cc:1467] Processing CreateTablet for tablet e0cf9a9235244c7394d6d66c6906fd1c (DEFAULT_TABLE table=test-table [id=9ea1a136903e430b96dcd1f3011fc52d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:56.221635 26640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0cf9a9235244c7394d6d66c6906fd1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:56.221983 26722 log.cc:824] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:56.222911 26720 raft_consensus.cc:348] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097d6337c7744808a706996c3b9c166b" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37093 } } peers { permanent_uuid: "986bf36105d3430a95595e8b4d34a3c3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35597 } } peers { permanent_uuid: "1cbe5133f9ef4cec88004434e981af05" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45361 } }
I20260619 02:49:56.223057 26720 raft_consensus.cc:374] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:56.223140 26720 raft_consensus.cc:729] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbe5133f9ef4cec88004434e981af05, State: Initialized, Role: FOLLOWER
I20260619 02:49:56.223269 26720 consensus_queue.cc:260] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097d6337c7744808a706996c3b9c166b" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37093 } } peers { permanent_uuid: "986bf36105d3430a95595e8b4d34a3c3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35597 } } peers { permanent_uuid: "1cbe5133f9ef4cec88004434e981af05" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45361 } }
I20260619 02:49:56.223564 26440 heartbeater.cc:503] Master 127.21.164.254:37543 was elected leader, sending a full tablet report...
I20260619 02:49:56.223547 26720 ts_tablet_manager.cc:1428] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 02:49:56.224593 26725 tablet_bootstrap.cc:492] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b: Bootstrap starting.
I20260619 02:49:56.225014 26722 tablet_bootstrap.cc:492] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: No bootstrap required, opened a new log
I20260619 02:49:56.225070 26722 ts_tablet_manager.cc:1397] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:49:56.225206 26725 tablet_bootstrap.cc:654] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:56.225646 26725 log.cc:824] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:56.226264 26725 tablet_bootstrap.cc:492] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b: No bootstrap required, opened a new log
I20260619 02:49:56.226330 26725 ts_tablet_manager.cc:1397] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:49:56.226624 26722 raft_consensus.cc:348] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097d6337c7744808a706996c3b9c166b" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37093 } } peers { permanent_uuid: "986bf36105d3430a95595e8b4d34a3c3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35597 } } peers { permanent_uuid: "1cbe5133f9ef4cec88004434e981af05" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45361 } }
I20260619 02:49:56.226752 26722 raft_consensus.cc:374] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:56.226773 26722 raft_consensus.cc:729] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 986bf36105d3430a95595e8b4d34a3c3, State: Initialized, Role: FOLLOWER
I20260619 02:49:56.226872 26722 consensus_queue.cc:260] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097d6337c7744808a706996c3b9c166b" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37093 } } peers { permanent_uuid: "986bf36105d3430a95595e8b4d34a3c3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35597 } } peers { permanent_uuid: "1cbe5133f9ef4cec88004434e981af05" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45361 } }
I20260619 02:49:56.227136 26722 ts_tablet_manager.cc:1428] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:56.227164 26573 heartbeater.cc:503] Master 127.21.164.254:37543 was elected leader, sending a full tablet report...
I20260619 02:49:56.228143 26725 raft_consensus.cc:348] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097d6337c7744808a706996c3b9c166b" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37093 } } peers { permanent_uuid: "986bf36105d3430a95595e8b4d34a3c3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35597 } } peers { permanent_uuid: "1cbe5133f9ef4cec88004434e981af05" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45361 } }
I20260619 02:49:56.228268 26725 raft_consensus.cc:374] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:56.228298 26725 raft_consensus.cc:729] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 097d6337c7744808a706996c3b9c166b, State: Initialized, Role: FOLLOWER
I20260619 02:49:56.228410 26725 consensus_queue.cc:260] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097d6337c7744808a706996c3b9c166b" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37093 } } peers { permanent_uuid: "986bf36105d3430a95595e8b4d34a3c3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35597 } } peers { permanent_uuid: "1cbe5133f9ef4cec88004434e981af05" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45361 } }
I20260619 02:49:56.228636 26725 ts_tablet_manager.cc:1428] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 02:49:56.228685 26706 heartbeater.cc:503] Master 127.21.164.254:37543 was elected leader, sending a full tablet report...
W20260619 02:49:56.294523 26574 tablet.cc:2369] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:56.346895 26724 raft_consensus.cc:482] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:56.347043 26724 raft_consensus.cc:504] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097d6337c7744808a706996c3b9c166b" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37093 } } peers { permanent_uuid: "986bf36105d3430a95595e8b4d34a3c3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35597 } } peers { permanent_uuid: "1cbe5133f9ef4cec88004434e981af05" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45361 } }
I20260619 02:49:56.347409 26724 leader_election.cc:288] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 097d6337c7744808a706996c3b9c166b (127.21.164.195:37093), 986bf36105d3430a95595e8b4d34a3c3 (127.21.164.194:35597)
I20260619 02:49:56.350191 26718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.347673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:44107 (local address 127.21.164.195:37093)
0619 02:49:56.347724 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.347726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.347739 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:49:56.347798 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.347799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.347809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.347883 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:56.348065 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.348535 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.349512 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.349885 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.349951 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.349960 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.350090 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.350125 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:49:56.350191 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.347622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:37093 (local address 127.21.164.193:44107)
0619 02:49:56.347678 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.347690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.347738 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.347932 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.347934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.347936 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:56.347992 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.347994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.348610 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.348611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.349416 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.349419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.349959 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.349960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.350032 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.350035 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.350079 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.350117 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 02:49:56.350322 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.347751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:35597 (local address 127.21.164.193:58333)
0619 02:49:56.347920 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.347928 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.347977 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.348101 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.348102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.348103 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:49:56.348168 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.348170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.348798 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.348799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.349549 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:56.349550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.350079 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.350080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:56.350157 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.350160 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.350204 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.350241 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260619 02:49:56.350319 26719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.347830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:58333 (local address 127.21.164.194:35597)
0619 02:49:56.347880 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.347882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.347984 (+ 102us) server_negotiation.cc:407] Connection header received
0619 02:49:56.348032 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.348033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.348042 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.348094 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:49:56.348236 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.348728 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.349636 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.350006 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.350069 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.350076 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.350217 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.350263 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:49:56.350562 26660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0cf9a9235244c7394d6d66c6906fd1c" candidate_uuid: "1cbe5133f9ef4cec88004434e981af05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "097d6337c7744808a706996c3b9c166b" is_pre_election: true
I20260619 02:49:56.350713 26660 raft_consensus.cc:2454] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbe5133f9ef4cec88004434e981af05 in term 0.
I20260619 02:49:56.350947 26327 leader_election.cc:302] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 097d6337c7744808a706996c3b9c166b, 1cbe5133f9ef4cec88004434e981af05; no voters:
I20260619 02:49:56.351038 26527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0cf9a9235244c7394d6d66c6906fd1c" candidate_uuid: "1cbe5133f9ef4cec88004434e981af05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "986bf36105d3430a95595e8b4d34a3c3" is_pre_election: true
I20260619 02:49:56.351095 26724 raft_consensus.cc:2781] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:56.351133 26724 raft_consensus.cc:482] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:56.351153 26724 raft_consensus.cc:3037] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:56.351190 26527 raft_consensus.cc:2454] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbe5133f9ef4cec88004434e981af05 in term 0.
I20260619 02:49:56.351889 26724 raft_consensus.cc:504] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097d6337c7744808a706996c3b9c166b" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37093 } } peers { permanent_uuid: "986bf36105d3430a95595e8b4d34a3c3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35597 } } peers { permanent_uuid: "1cbe5133f9ef4cec88004434e981af05" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45361 } }
I20260619 02:49:56.352005 26724 leader_election.cc:288] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [CANDIDATE]: Term 1 election: Requested vote from peers 097d6337c7744808a706996c3b9c166b (127.21.164.195:37093), 986bf36105d3430a95595e8b4d34a3c3 (127.21.164.194:35597)
I20260619 02:49:56.352167 26660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0cf9a9235244c7394d6d66c6906fd1c" candidate_uuid: "1cbe5133f9ef4cec88004434e981af05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "097d6337c7744808a706996c3b9c166b"
I20260619 02:49:56.352190 26527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0cf9a9235244c7394d6d66c6906fd1c" candidate_uuid: "1cbe5133f9ef4cec88004434e981af05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "986bf36105d3430a95595e8b4d34a3c3"
I20260619 02:49:56.352240 26660 raft_consensus.cc:3037] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:56.352254 26527 raft_consensus.cc:3037] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:56.353001 26660 raft_consensus.cc:2454] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbe5133f9ef4cec88004434e981af05 in term 1.
I20260619 02:49:56.353009 26527 raft_consensus.cc:2454] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbe5133f9ef4cec88004434e981af05 in term 1.
I20260619 02:49:56.353168 26327 leader_election.cc:302] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbe5133f9ef4cec88004434e981af05, 986bf36105d3430a95595e8b4d34a3c3; no voters:
I20260619 02:49:56.353305 26724 raft_consensus.cc:2781] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:56.353485 26724 raft_consensus.cc:686] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbe5133f9ef4cec88004434e981af05, State: Running, Role: LEADER
I20260619 02:49:56.353585 26724 consensus_queue.cc:237] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097d6337c7744808a706996c3b9c166b" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37093 } } peers { permanent_uuid: "986bf36105d3430a95595e8b4d34a3c3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35597 } } peers { permanent_uuid: "1cbe5133f9ef4cec88004434e981af05" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45361 } }
I20260619 02:49:56.354470 26250 catalog_manager.cc:5495] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbe5133f9ef4cec88004434e981af05 (127.21.164.193). New cstate: current_term: 1 leader_uuid: "1cbe5133f9ef4cec88004434e981af05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097d6337c7744808a706996c3b9c166b" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "986bf36105d3430a95595e8b4d34a3c3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbe5133f9ef4cec88004434e981af05" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45361 } health_report { overall_health: HEALTHY } } }
W20260619 02:49:56.393393 26441 tablet.cc:2369] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:56.395524 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.393045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.21.164.193:45361)
0619 02:49:56.393102 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.393105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.393112 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:49:56.393197 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.393198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.393216 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.393279 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:49:56.393497 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.394031 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.394760 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:56.394957 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:56.395010 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:56.395186 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:49:56.395285 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:49:56.395350 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.395408 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.395426 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:56.397233 26660 raft_consensus.cc:1261] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b [term 1 FOLLOWER]: Refusing update from remote peer 1cbe5133f9ef4cec88004434e981af05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:56.397233 26527 raft_consensus.cc:1261] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3 [term 1 FOLLOWER]: Refusing update from remote peer 1cbe5133f9ef4cec88004434e981af05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:56.397549 26724 consensus_queue.cc:1035] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "986bf36105d3430a95595e8b4d34a3c3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:56.397688 26730 consensus_queue.cc:1035] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "097d6337c7744808a706996c3b9c166b" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:56.400288 26734 mvcc.cc:205] Tried to move back new op lower bound from 7298405975640346624 to 7298405975464980480. Current Snapshot: MvccSnapshot[applied={T|T < 7298405975640346624}]
I20260619 02:49:56.400089 26733 mvcc.cc:205] Tried to move back new op lower bound from 7298405975640346624 to 7298405975464980480. Current Snapshot: MvccSnapshot[applied={T|T < 7298405975640346624}]
I20260619 02:49:56.400847 26735 mvcc.cc:205] Tried to move back new op lower bound from 7298405975640346624 to 7298405975464980480. Current Snapshot: MvccSnapshot[applied={T|T < 7298405975640346624}]
I20260619 02:49:56.402915 26249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:56.403003 26249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:56.403743 26249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3502f954b024448f9301863e2adf469e: Sending DeleteTablet for 3 replicas of tablet e0cf9a9235244c7394d6d66c6906fd1c
I20260619 02:49:56.404047 26374 tablet_service.cc:1514] Processing DeleteTablet for tablet e0cf9a9235244c7394d6d66c6906fd1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:56 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:33708
I20260619 02:49:56.404227 26507 tablet_service.cc:1514] Processing DeleteTablet for tablet e0cf9a9235244c7394d6d66c6906fd1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:56 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:51506
I20260619 02:49:56.404256 26640 tablet_service.cc:1514] Processing DeleteTablet for tablet e0cf9a9235244c7394d6d66c6906fd1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:56 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:35010
I20260619 02:49:56.404464 26744 tablet_replica.cc:330] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b: stopping tablet replica
I20260619 02:49:56.404533 26742 tablet_replica.cc:330] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05: stopping tablet replica
I20260619 02:49:56.404631 26742 raft_consensus.cc:2229] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:56.404793 26742 raft_consensus.cc:2258] T e0cf9a9235244c7394d6d66c6906fd1c P 1cbe5133f9ef4cec88004434e981af05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:56.404997 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 26312
I20260619 02:49:56.406158 26744 raft_consensus.cc:2229] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:56.406286 26744 raft_consensus.cc:2258] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:56.406455 26743 tablet_replica.cc:330] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: stopping tablet replica
I20260619 02:49:56.406565 26743 raft_consensus.cc:2229] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:56.406673 26743 raft_consensus.cc:2258] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:56.406843 26744 ts_tablet_manager.cc:1905] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:56.407111 26743 ts_tablet_manager.cc:1905] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:56.408614 26744 ts_tablet_manager.cc:1918] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:56.408633 26743 ts_tablet_manager.cc:1918] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:56.408715 26743 log.cc:1196] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/e0cf9a9235244c7394d6d66c6906fd1c
I20260619 02:49:56.408821 26744 log.cc:1196] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/e0cf9a9235244c7394d6d66c6906fd1c
I20260619 02:49:56.408975 26743 ts_tablet_manager.cc:1939] T e0cf9a9235244c7394d6d66c6906fd1c P 986bf36105d3430a95595e8b4d34a3c3: Deleting consensus metadata
I20260619 02:49:56.409159 26744 ts_tablet_manager.cc:1939] T e0cf9a9235244c7394d6d66c6906fd1c P 097d6337c7744808a706996c3b9c166b: Deleting consensus metadata
I20260619 02:49:56.409404 26234 catalog_manager.cc:4841] TS 986bf36105d3430a95595e8b4d34a3c3 (127.21.164.194:35597): tablet e0cf9a9235244c7394d6d66c6906fd1c (table test-table [id=9ea1a136903e430b96dcd1f3011fc52d]) successfully deleted
I20260619 02:49:56.409611 26234 catalog_manager.cc:4841] TS 097d6337c7744808a706996c3b9c166b (127.21.164.195:37093): tablet e0cf9a9235244c7394d6d66c6906fd1c (table test-table [id=9ea1a136903e430b96dcd1f3011fc52d]) successfully deleted
W20260619 02:49:56.409737 26236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:45361 (error 108)
I20260619 02:49:56.409989 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 26445
I20260619 02:49:56.410145 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.410041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:45361 (local address 127.0.0.1:33732)
0619 02:49:56.410087 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.410115 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:45361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 02:49:56.410600 26236 catalog_manager.cc:4584] TS 1cbe5133f9ef4cec88004434e981af05 (127.21.164.193:45361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0cf9a9235244c7394d6d66c6906fd1c: Network error: Client connection negotiation failed: client connection to 127.21.164.193:45361: connect: Connection refused (error 111)
I20260619 02:49:56.414693 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 26578
I20260619 02:49:56.418674 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 26219
2026-06-19T02:49:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1913 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T02:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:56Z Disabled control of system clock
I20260619 02:49:56.443763 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:35527
--webserver_interface=127.21.164.254
--webserver_port=0
--builtin_ntp_servers=127.21.164.212:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:35527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:49:56.524120 26749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:56.524348 26749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:56.524369 26749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:56.525540 26749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:56.525583 26749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:56.525593 26749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:56.525604 26749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:56.525615 26749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:56.526902 26749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:35527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:35527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:56.527107 26749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:56.527402 26749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:56.532025 26757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:56.532043 26755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:56.532088 26754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:56.532120 26749 server_base.cc:1034] running on GCE node
I20260619 02:49:56.532514 26749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:56.532823 26749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:56.533973 26749 hybrid_clock.cc:648] HybridClock initialized: now 1781837396533957 us; error 30 us; skew 500 ppm
I20260619 02:49:56.535218 26749 webserver.cc:460] Webserver started at http://127.21.164.254:38069/ using document root <none> and password file <none>
I20260619 02:49:56.535442 26749 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:56.535493 26749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:56.535616 26749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:56.536718 26749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "2884c2083b014a43bcb683dadc3d6f77"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "2e424f7813aa4f936856ebea6e79526d"
server_key_iv: "31c584d7156afc74f03528fa471b5ff6"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.537058 26749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "2884c2083b014a43bcb683dadc3d6f77"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "2e424f7813aa4f936856ebea6e79526d"
server_key_iv: "31c584d7156afc74f03528fa471b5ff6"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.538364 26749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:56.539036 26763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.539181 26749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:56.539261 26749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "2884c2083b014a43bcb683dadc3d6f77"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "2e424f7813aa4f936856ebea6e79526d"
server_key_iv: "31c584d7156afc74f03528fa471b5ff6"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.539326 26749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:56.542670 26749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:56.542924 26749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:56.543041 26749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:56.546588 26749 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:35527
I20260619 02:49:56.546617 26815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:35527 every 8 connection(s)
I20260619 02:49:56.546926 26749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:56.547567 26816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:56.547596 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 26749
I20260619 02:49:56.547689 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:56.547869 22163 external_mini_cluster.cc:1428] Setting key 04686552398065b9427cc1c044537847
I20260619 02:49:56.550314 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.548382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.21.164.254:35527)
0619 02:49:56.548569 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.548577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.548630 (+ 53us) server_negotiation.cc:407] Connection header received
0619 02:49:56.549085 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.549092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.549133 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.549268 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.549924 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:56.549927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:56.549928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:56.549932 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:56.549971 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:56.549981 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:56.550038 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.550104 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.550160 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260619 02:49:56.551082 26816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77: Bootstrap starting.
I20260619 02:49:56.551652 26816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:56.551901 26816 log.cc:824] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:56.552536 26816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77: No bootstrap required, opened a new log
I20260619 02:49:56.553841 26816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884c2083b014a43bcb683dadc3d6f77" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 35527 } }
I20260619 02:49:56.553977 26816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:56.554008 26816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2884c2083b014a43bcb683dadc3d6f77, State: Initialized, Role: FOLLOWER
I20260619 02:49:56.554121 26816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884c2083b014a43bcb683dadc3d6f77" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 35527 } }
I20260619 02:49:56.554203 26816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:56.554234 26816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:56.554279 26816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:56.554765 26816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884c2083b014a43bcb683dadc3d6f77" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 35527 } }
I20260619 02:49:56.554859 26816 leader_election.cc:302] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2884c2083b014a43bcb683dadc3d6f77; no voters:
I20260619 02:49:56.555007 26816 leader_election.cc:288] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:56.555253 26816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:56.555270 26821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:56.555449 26821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [term 1 LEADER]: Becoming Leader. State: Replica: 2884c2083b014a43bcb683dadc3d6f77, State: Running, Role: LEADER
I20260619 02:49:56.555531 26821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884c2083b014a43bcb683dadc3d6f77" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 35527 } }
I20260619 02:49:56.555907 26823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2884c2083b014a43bcb683dadc3d6f77. Latest consensus state: current_term: 1 leader_uuid: "2884c2083b014a43bcb683dadc3d6f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884c2083b014a43bcb683dadc3d6f77" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 35527 } } }
I20260619 02:49:56.555927 26822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2884c2083b014a43bcb683dadc3d6f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2884c2083b014a43bcb683dadc3d6f77" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 35527 } } }
I20260619 02:49:56.556102 26823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:56.556108 26822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:56.556316 26830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:56.556808 26830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:56.558257 26830 catalog_manager.cc:1347] Generated new cluster ID: 7c208309b66e4ae6af138c9ab6aff3ba
I20260619 02:49:56.558312 26830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:56.572503 26830 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:56.572710 26830 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:56.576203 26830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77: Generated new TSK 0
I20260619 02:49:56.576411 26830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:56.583974 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:35527
--builtin_ntp_servers=127.21.164.212:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:49:56.664530 26840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:56.664784 26840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:56.664811 26840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:56.666039 26840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:56.666087 26840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:56.666113 26840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:49:56.667479 26840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:35527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:56.667807 26840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:56.668150 26840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:56.673132 26848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:56.673251 26840 server_base.cc:1034] running on GCE node
W20260619 02:49:56.673141 26845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:56.673149 26846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:56.673589 26840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:56.673827 26840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:56.674978 26840 hybrid_clock.cc:648] HybridClock initialized: now 1781837396674966 us; error 35 us; skew 500 ppm
I20260619 02:49:56.676357 26840 webserver.cc:460] Webserver started at http://127.21.164.193:35097/ using document root <none> and password file <none>
I20260619 02:49:56.676625 26840 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:56.676714 26840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:56.676844 26840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:56.677917 26840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "568ab555cbb6480b8cfdd9b0316785e8"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "90088a6fe09bb303ae857645a0cf4108"
server_key_iv: "987cea0bf02887d2a1e362df10278f1d"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.678267 26840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "568ab555cbb6480b8cfdd9b0316785e8"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "90088a6fe09bb303ae857645a0cf4108"
server_key_iv: "987cea0bf02887d2a1e362df10278f1d"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.679497 26840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:49:56.680204 26854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.680361 26840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:56.680472 26840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "568ab555cbb6480b8cfdd9b0316785e8"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "90088a6fe09bb303ae857645a0cf4108"
server_key_iv: "987cea0bf02887d2a1e362df10278f1d"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.680538 26840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:56.683909 26840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:56.684183 26840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:56.684309 26840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:56.684499 26840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:56.684830 26840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:56.684865 26840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.684893 26840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:56.684916 26840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.690374 26840 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:36177
I20260619 02:49:56.690460 26967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:36177 every 8 connection(s)
I20260619 02:49:56.690758 26840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:56.693651 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.691493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:48529 (local address 127.21.164.254:35527)
0619 02:49:56.691563 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.691565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.691757 (+ 192us) server_negotiation.cc:407] Connection header received
0619 02:49:56.692242 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.692243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.692250 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.692305 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.693465 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:56.693467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:56.693467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:56.693471 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:56.693487 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:56.693491 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:56.693533 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.693584 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.693595 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:49:56.693835 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.691466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:35527 (local address 127.21.164.193:48529)
0619 02:49:56.691639 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.691677 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.692134 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.692365 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.692368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.692376 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:56.693328 (+ 952us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:56.693328 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:56.693358 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:56.693387 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:56.693532 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:56.693535 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:56.693537 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.693589 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.693639 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260619 02:49:56.694293 26968 heartbeater.cc:347] Connected to a master server at 127.21.164.254:35527
I20260619 02:49:56.694375 26968 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:56.694507 26968 heartbeater.cc:511] Master 127.21.164.254:35527 requested a full tablet report, sending...
I20260619 02:49:56.694970 26780 ts_manager.cc:194] Registered new tserver with Master: 568ab555cbb6480b8cfdd9b0316785e8 (127.21.164.193:36177)
I20260619 02:49:56.698593 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 26840
I20260619 02:49:56.698686 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:56.698859 22163 external_mini_cluster.cc:1428] Setting key ba22a045cab1992984af5c6f8ae56b22
I20260619 02:49:56.699687 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:35527
--builtin_ntp_servers=127.21.164.212:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:49:56.778214 26971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:56.778441 26971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:56.778463 26971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:56.779633 26971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:56.779675 26971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:56.779700 26971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:49:56.781076 26971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:35527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:56.781307 26971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:56.781631 26971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:56.786558 26979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:56.786582 26977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:56.786554 26976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:56.786931 26971 server_base.cc:1034] running on GCE node
I20260619 02:49:56.787143 26971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:56.787374 26971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:56.788535 26971 hybrid_clock.cc:648] HybridClock initialized: now 1781837396788522 us; error 26 us; skew 500 ppm
I20260619 02:49:56.789709 26971 webserver.cc:460] Webserver started at http://127.21.164.194:37155/ using document root <none> and password file <none>
I20260619 02:49:56.789938 26971 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:56.789983 26971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:56.790108 26971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:56.791189 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "633b9d0d127e47e591062c2523964a85"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "e8197708c7301146e9dcca42c6448e55"
server_key_iv: "214fb92e541eaaf2dbec46587031cc34"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.791560 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "633b9d0d127e47e591062c2523964a85"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "e8197708c7301146e9dcca42c6448e55"
server_key_iv: "214fb92e541eaaf2dbec46587031cc34"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.792825 26971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:49:56.793478 26985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.793663 26971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:56.793747 26971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "633b9d0d127e47e591062c2523964a85"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "e8197708c7301146e9dcca42c6448e55"
server_key_iv: "214fb92e541eaaf2dbec46587031cc34"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.793828 26971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:56.797124 26971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:56.797381 26971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:56.797499 26971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:56.797691 26971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:56.797977 26971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:56.798008 26971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.798034 26971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:56.798058 26971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.803411 26971 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:35669
I20260619 02:49:56.803448 27098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:35669 every 8 connection(s)
I20260619 02:49:56.803759 26971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:49:56.803941 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 26971
I20260619 02:49:56.804035 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:56.804209 22163 external_mini_cluster.cc:1428] Setting key c2335d22ed1a3b6cc3f6e068ec6ea47f
I20260619 02:49:56.805154 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:35527
--builtin_ntp_servers=127.21.164.212:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 02:49:56.806710 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.804470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:54209 (local address 127.21.164.254:35527)
0619 02:49:56.804517 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.804519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.804840 (+ 321us) server_negotiation.cc:407] Connection header received
0619 02:49:56.805267 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.805268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.805274 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.805327 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.806477 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:56.806478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:56.806478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:56.806482 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:56.806498 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:56.806502 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:56.806553 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.806626 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.806639 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:49:56.806901 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.804463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:35527 (local address 127.21.164.194:54209)
0619 02:49:56.804635 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.804693 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.805171 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.805403 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.805407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.805414 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:56.806333 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:56.806333 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:56.806362 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:56.806392 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:56.806556 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:56.806561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:56.806564 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.806632 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.806697 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260619 02:49:56.807372 27099 heartbeater.cc:347] Connected to a master server at 127.21.164.254:35527
I20260619 02:49:56.807457 27099 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:56.807595 27099 heartbeater.cc:511] Master 127.21.164.254:35527 requested a full tablet report, sending...
I20260619 02:49:56.807937 26780 ts_manager.cc:194] Registered new tserver with Master: 633b9d0d127e47e591062c2523964a85 (127.21.164.194:35669)
W20260619 02:49:56.884557 27102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:56.884790 27102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:56.884812 27102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:56.885984 27102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:56.886027 27102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:56.886052 27102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:49:56.887382 27102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:35527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:56.887625 27102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:56.887910 27102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:56.892721 27108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:56.892719 27110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:56.892726 27107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:56.892773 27102 server_base.cc:1034] running on GCE node
I20260619 02:49:56.893086 27102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:56.893303 27102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:56.894446 27102 hybrid_clock.cc:648] HybridClock initialized: now 1781837396894434 us; error 28 us; skew 500 ppm
I20260619 02:49:56.895552 27102 webserver.cc:460] Webserver started at http://127.21.164.195:37001/ using document root <none> and password file <none>
I20260619 02:49:56.895758 27102 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:56.895807 27102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:56.895923 27102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:56.897060 27102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "547346da43564942a69666faed3c9d92"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "1d7d7bf3b6038de591aea9a02bcbbd93"
server_key_iv: "9da00041f9a4115d74a5ef9cb29bca62"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.897399 27102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "547346da43564942a69666faed3c9d92"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "1d7d7bf3b6038de591aea9a02bcbbd93"
server_key_iv: "9da00041f9a4115d74a5ef9cb29bca62"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.898592 27102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:56.899235 27116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.899367 27102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:56.899448 27102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "547346da43564942a69666faed3c9d92"
format_stamp: "Formatted at 2026-06-19 02:49:56 on dist-test-slave-jt0t"
server_key: "1d7d7bf3b6038de591aea9a02bcbbd93"
server_key_iv: "9da00041f9a4115d74a5ef9cb29bca62"
server_key_version: "encryptionkey@0"
I20260619 02:49:56.899506 27102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:56.902753 27102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:56.902997 27102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:56.903116 27102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:56.903316 27102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:56.903595 27102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:56.903627 27102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.903654 27102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:56.903677 27102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:56.909004 27102 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:42639
I20260619 02:49:56.909041 27229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:42639 every 8 connection(s)
I20260619 02:49:56.909368 27102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:49:56.909821 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 27102
I20260619 02:49:56.909926 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:56.910113 22163 external_mini_cluster.cc:1428] Setting key 375751d99c29a7cfbb84838a01e197b9
I20260619 02:49:56.912299 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.910059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:33149 (local address 127.21.164.254:35527)
0619 02:49:56.910106 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.910108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.910443 (+ 335us) server_negotiation.cc:407] Connection header received
0619 02:49:56.911027 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.911028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.911032 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.911064 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.912117 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:56.912118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:56.912118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:56.912122 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:56.912137 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:56.912140 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:56.912183 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.912230 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.912241 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:49:56.912499 27232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.910072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:35527 (local address 127.21.164.195:33149)
0619 02:49:56.910288 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.910329 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.910932 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.911129 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.911132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.911140 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:56.911977 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:56.911978 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:56.912006 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:56.912034 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:56.912184 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:56.912186 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:56.912189 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.912237 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.912283 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260619 02:49:56.912905 27230 heartbeater.cc:347] Connected to a master server at 127.21.164.254:35527
I20260619 02:49:56.912981 27230 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:56.913118 27230 heartbeater.cc:511] Master 127.21.164.254:35527 requested a full tablet report, sending...
I20260619 02:49:56.913441 26780 ts_manager.cc:194] Registered new tserver with Master: 547346da43564942a69666faed3c9d92 (127.21.164.195:42639)
I20260619 02:49:56.921868 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:49:56.924134 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.923196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.21.164.254:35527)
0619 02:49:56.923263 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.923265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.923346 (+ 81us) server_negotiation.cc:407] Connection header received
0619 02:49:56.923401 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.923402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.923408 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.923454 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.923955 (+ 501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:56.923956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:56.923957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:56.923960 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:56.923974 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:56.923977 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:56.924015 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.924051 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.924062 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:49:56.925649 26780 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:49:56.927151 26780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:49:56.930475 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.929146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:42639 (local address 127.0.0.1:44794)
0619 02:49:56.929385 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.929396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.929439 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.929576 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.929580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.929596 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:56.930072 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:56.930073 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:56.930090 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:56.930114 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:56.930319 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:56.930325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:56.930328 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.930391 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.930405 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":85,"threads_started":1}
I20260619 02:49:56.930512 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.929278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.21.164.195:42639)
0619 02:49:56.929457 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.929461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.929472 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:49:56.929495 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.929497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.929504 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.929564 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.930180 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:56.930187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:56.930188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:56.930192 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:56.930224 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:56.930244 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:56.930294 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.930409 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.930427 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260619 02:49:56.931413 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.929118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:36177 (local address 127.0.0.1:43462)
0619 02:49:56.929295 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.929336 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.929392 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.930109 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.930109 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.930111 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:56.930991 (+ 880us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:56.930992 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:56.930996 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:56.931003 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:56.931288 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:56.931289 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:56.931290 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.931336 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.931349 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260619 02:49:56.931326 27164 tablet_service.cc:1467] Processing CreateTablet for tablet 23a9f75a1f3a48588fc0ab9163f97125 (DEFAULT_TABLE table=test-table [id=e86b6052230e4ee3bdc0be091617023e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:56.931697 27164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a9f75a1f3a48588fc0ab9163f97125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:56.931710 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.929216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.21.164.193:36177)
0619 02:49:56.929704 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.929709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.929718 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:56.929743 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.929745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.929753 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.929823 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.931076 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:56.931088 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:56.931088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:56.931093 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:56.931146 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:56.931182 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:56.931260 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.931379 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.931396 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":77,"threads_started":1}
I20260619 02:49:56.934314 27244 tablet_bootstrap.cc:492] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92: Bootstrap starting.
I20260619 02:49:56.934868 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.929364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:35669 (local address 127.0.0.1:56326)
0619 02:49:56.929549 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:56.929557 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:56.929595 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:56.930107 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:56.930108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:56.930111 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:56.930894 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:56.930895 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:56.930901 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:56.930913 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:56.934745 (+ 3832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:56.934746 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:56.934747 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:49:56.934775 (+ 28us) client_negotiation.cc:240] Negotiation successful
0619 02:49:56.934799 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":57,"threads_started":1}
I20260619 02:49:56.935007 27244 tablet_bootstrap.cc:654] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:56.934907 26902 tablet_service.cc:1467] Processing CreateTablet for tablet 23a9f75a1f3a48588fc0ab9163f97125 (DEFAULT_TABLE table=test-table [id=e86b6052230e4ee3bdc0be091617023e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:56.935175 27243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:56.929532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.21.164.194:35669)
0619 02:49:56.929706 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:56.929709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:56.929718 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:49:56.929742 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:56.929744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:56.929753 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:56.929831 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:56.934544 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:56.934558 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:56.934558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:56.934565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:56.934611 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:56.934652 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:56.935046 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:56.935074 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 02:49:56.935091 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260619 02:49:56.935245 26902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a9f75a1f3a48588fc0ab9163f97125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:56.935331 27244 log.cc:824] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:56.935740 27033 tablet_service.cc:1467] Processing CreateTablet for tablet 23a9f75a1f3a48588fc0ab9163f97125 (DEFAULT_TABLE table=test-table [id=e86b6052230e4ee3bdc0be091617023e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:49:56.936066 27033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23a9f75a1f3a48588fc0ab9163f97125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:56.936115 27244 tablet_bootstrap.cc:492] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92: No bootstrap required, opened a new log
I20260619 02:49:56.936168 27244 ts_tablet_manager.cc:1397] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:49:56.938131 27246 tablet_bootstrap.cc:492] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8: Bootstrap starting.
I20260619 02:49:56.938925 27246 tablet_bootstrap.cc:654] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:56.938699 27244 raft_consensus.cc:348] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568ab555cbb6480b8cfdd9b0316785e8" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36177 } } peers { permanent_uuid: "547346da43564942a69666faed3c9d92" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 42639 } } peers { permanent_uuid: "633b9d0d127e47e591062c2523964a85" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35669 } }
I20260619 02:49:56.939000 27244 raft_consensus.cc:374] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:56.939055 27244 raft_consensus.cc:729] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547346da43564942a69666faed3c9d92, State: Initialized, Role: FOLLOWER
I20260619 02:49:56.939196 27244 consensus_queue.cc:260] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568ab555cbb6480b8cfdd9b0316785e8" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36177 } } peers { permanent_uuid: "547346da43564942a69666faed3c9d92" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 42639 } } peers { permanent_uuid: "633b9d0d127e47e591062c2523964a85" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35669 } }
I20260619 02:49:56.939256 27246 log.cc:824] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:56.939455 27244 ts_tablet_manager.cc:1428] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 02:49:56.939481 27247 tablet_bootstrap.cc:492] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: Bootstrap starting.
I20260619 02:49:56.939581 27230 heartbeater.cc:503] Master 127.21.164.254:35527 was elected leader, sending a full tablet report...
I20260619 02:49:56.940058 27246 tablet_bootstrap.cc:492] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8: No bootstrap required, opened a new log
I20260619 02:49:56.940096 27247 tablet_bootstrap.cc:654] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:56.940208 27246 ts_tablet_manager.cc:1397] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:49:56.940452 27247 log.cc:824] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:56.941246 27247 tablet_bootstrap.cc:492] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: No bootstrap required, opened a new log
I20260619 02:49:56.941355 27247 ts_tablet_manager.cc:1397] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 02:49:56.942888 27246 raft_consensus.cc:348] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568ab555cbb6480b8cfdd9b0316785e8" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36177 } } peers { permanent_uuid: "547346da43564942a69666faed3c9d92" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 42639 } } peers { permanent_uuid: "633b9d0d127e47e591062c2523964a85" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35669 } }
I20260619 02:49:56.943064 27246 raft_consensus.cc:374] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:56.943109 27246 raft_consensus.cc:729] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568ab555cbb6480b8cfdd9b0316785e8, State: Initialized, Role: FOLLOWER
I20260619 02:49:56.943233 27246 consensus_queue.cc:260] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568ab555cbb6480b8cfdd9b0316785e8" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36177 } } peers { permanent_uuid: "547346da43564942a69666faed3c9d92" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 42639 } } peers { permanent_uuid: "633b9d0d127e47e591062c2523964a85" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35669 } }
I20260619 02:49:56.943528 27246 ts_tablet_manager.cc:1428] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:49:56.943535 26968 heartbeater.cc:503] Master 127.21.164.254:35527 was elected leader, sending a full tablet report...
I20260619 02:49:56.943863 27247 raft_consensus.cc:348] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568ab555cbb6480b8cfdd9b0316785e8" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36177 } } peers { permanent_uuid: "547346da43564942a69666faed3c9d92" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 42639 } } peers { permanent_uuid: "633b9d0d127e47e591062c2523964a85" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35669 } }
I20260619 02:49:56.944010 27247 raft_consensus.cc:374] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:56.944042 27247 raft_consensus.cc:729] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633b9d0d127e47e591062c2523964a85, State: Initialized, Role: FOLLOWER
I20260619 02:49:56.944151 27247 consensus_queue.cc:260] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568ab555cbb6480b8cfdd9b0316785e8" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36177 } } peers { permanent_uuid: "547346da43564942a69666faed3c9d92" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 42639 } } peers { permanent_uuid: "633b9d0d127e47e591062c2523964a85" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35669 } }
I20260619 02:49:56.944396 27247 ts_tablet_manager.cc:1428] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 02:49:56.944422 27099 heartbeater.cc:503] Master 127.21.164.254:35527 was elected leader, sending a full tablet report...
W20260619 02:49:57.054469 27100 tablet.cc:2369] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:49:57.079720 27252 raft_consensus.cc:482] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:49:57.079865 27252 raft_consensus.cc:504] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568ab555cbb6480b8cfdd9b0316785e8" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36177 } } peers { permanent_uuid: "547346da43564942a69666faed3c9d92" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 42639 } } peers { permanent_uuid: "633b9d0d127e47e591062c2523964a85" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35669 } }
I20260619 02:49:57.080200 27252 leader_election.cc:288] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 568ab555cbb6480b8cfdd9b0316785e8 (127.21.164.193:36177), 547346da43564942a69666faed3c9d92 (127.21.164.195:42639)
I20260619 02:49:57.086820 27240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.080475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:59663 (local address 127.21.164.193:36177)
0619 02:49:57.080536 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.080539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.080716 (+ 177us) server_negotiation.cc:407] Connection header received
0619 02:49:57.080805 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.080806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.080812 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.080854 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.086581 (+ 5727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:57.086582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:57.086583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:57.086588 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:57.086605 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:57.086609 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:57.086661 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:57.086717 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 02:49:57.086730 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:49:57.086817 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.080465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:46247 (local address 127.21.164.195:42639)
0619 02:49:57.080591 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.080592 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.080600 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:57.080640 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.080641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.080665 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.080710 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.086595 (+ 5885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:57.086596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:57.086597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:57.086602 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:57.086618 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:57.086623 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:57.086670 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:57.086723 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 02:49:57.086736 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 02:49:57.086824 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.080403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:42639 (local address 127.21.164.194:46247)
0619 02:49:57.080475 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:57.080486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:57.080527 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:57.080733 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:57.080734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:57.080737 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:57.086476 (+ 5739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:57.086476 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:57.086484 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:57.086500 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:57.086670 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:57.086671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:57.086672 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:49:57.086720 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:49:57.086735 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 02:49:57.086846 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.080407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:36177 (local address 127.21.164.194:59663)
0619 02:49:57.080615 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:57.080625 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:57.080739 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:57.080875 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:57.080876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:57.080878 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:57.086430 (+ 5552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:49:57.086431 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:57.086442 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:49:57.086462 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:57.086662 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:57.086663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:57.086664 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:49:57.086712 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:49:57.086728 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":61,"thread_start_us":95,"threads_started":1}
I20260619 02:49:57.087211 26922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a9f75a1f3a48588fc0ab9163f97125" candidate_uuid: "633b9d0d127e47e591062c2523964a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568ab555cbb6480b8cfdd9b0316785e8" is_pre_election: true
I20260619 02:49:57.087210 27184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a9f75a1f3a48588fc0ab9163f97125" candidate_uuid: "633b9d0d127e47e591062c2523964a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "547346da43564942a69666faed3c9d92" is_pre_election: true
I20260619 02:49:57.087376 27184 raft_consensus.cc:2454] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633b9d0d127e47e591062c2523964a85 in term 0.
I20260619 02:49:57.087376 26922 raft_consensus.cc:2454] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633b9d0d127e47e591062c2523964a85 in term 0.
I20260619 02:49:57.087661 26989 leader_election.cc:302] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547346da43564942a69666faed3c9d92, 633b9d0d127e47e591062c2523964a85; no voters:
I20260619 02:49:57.087797 27252 raft_consensus.cc:2781] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:49:57.087850 27252 raft_consensus.cc:482] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:49:57.087867 27252 raft_consensus.cc:3037] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:57.088546 27252 raft_consensus.cc:504] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568ab555cbb6480b8cfdd9b0316785e8" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36177 } } peers { permanent_uuid: "547346da43564942a69666faed3c9d92" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 42639 } } peers { permanent_uuid: "633b9d0d127e47e591062c2523964a85" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35669 } }
I20260619 02:49:57.088697 27252 leader_election.cc:288] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [CANDIDATE]: Term 1 election: Requested vote from peers 568ab555cbb6480b8cfdd9b0316785e8 (127.21.164.193:36177), 547346da43564942a69666faed3c9d92 (127.21.164.195:42639)
I20260619 02:49:57.088858 26922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a9f75a1f3a48588fc0ab9163f97125" candidate_uuid: "633b9d0d127e47e591062c2523964a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568ab555cbb6480b8cfdd9b0316785e8"
I20260619 02:49:57.088868 27184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23a9f75a1f3a48588fc0ab9163f97125" candidate_uuid: "633b9d0d127e47e591062c2523964a85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "547346da43564942a69666faed3c9d92"
I20260619 02:49:57.088948 27184 raft_consensus.cc:3037] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:57.088949 26922 raft_consensus.cc:3037] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:57.089828 27184 raft_consensus.cc:2454] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633b9d0d127e47e591062c2523964a85 in term 1.
I20260619 02:49:57.089833 26922 raft_consensus.cc:2454] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633b9d0d127e47e591062c2523964a85 in term 1.
I20260619 02:49:57.090016 26989 leader_election.cc:302] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547346da43564942a69666faed3c9d92, 633b9d0d127e47e591062c2523964a85; no voters:
I20260619 02:49:57.090106 27252 raft_consensus.cc:2781] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:57.090245 27252 raft_consensus.cc:686] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 1 LEADER]: Becoming Leader. State: Replica: 633b9d0d127e47e591062c2523964a85, State: Running, Role: LEADER
I20260619 02:49:57.090337 27252 consensus_queue.cc:237] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568ab555cbb6480b8cfdd9b0316785e8" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36177 } } peers { permanent_uuid: "547346da43564942a69666faed3c9d92" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 42639 } } peers { permanent_uuid: "633b9d0d127e47e591062c2523964a85" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35669 } }
I20260619 02:49:57.091199 26780 catalog_manager.cc:5495] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 633b9d0d127e47e591062c2523964a85 (127.21.164.194). New cstate: current_term: 1 leader_uuid: "633b9d0d127e47e591062c2523964a85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568ab555cbb6480b8cfdd9b0316785e8" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "547346da43564942a69666faed3c9d92" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "633b9d0d127e47e591062c2523964a85" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 35669 } health_report { overall_health: HEALTHY } } }
I20260619 02:49:57.113658 27243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.112749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.21.164.194:35669)
0619 02:49:57.112809 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.112811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.112819 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:49:57.112855 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.112856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.112870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.112922 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.113462 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:57.113463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:57.113463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:57.113467 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:57.113480 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:57.113484 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:57.113528 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:57.113571 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 02:49:57.113582 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:49:57.115406 26922 raft_consensus.cc:1261] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8 [term 1 FOLLOWER]: Refusing update from remote peer 633b9d0d127e47e591062c2523964a85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:57.115412 27184 raft_consensus.cc:1261] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92 [term 1 FOLLOWER]: Refusing update from remote peer 633b9d0d127e47e591062c2523964a85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:49:57.115762 27254 consensus_queue.cc:1035] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "547346da43564942a69666faed3c9d92" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:57.115890 27252 consensus_queue.cc:1035] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "568ab555cbb6480b8cfdd9b0316785e8" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:49:57.118161 27257 mvcc.cc:205] Tried to move back new op lower bound from 7298405978582077440 to 7298405978482638848. Current Snapshot: MvccSnapshot[applied={T|T < 7298405978582077440}]
I20260619 02:49:57.118979 27258 mvcc.cc:205] Tried to move back new op lower bound from 7298405978582077440 to 7298405978482638848. Current Snapshot: MvccSnapshot[applied={T|T < 7298405978582077440}]
I20260619 02:49:57.120337 26779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:57.120453 26779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:49:57.121209 26779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2884c2083b014a43bcb683dadc3d6f77: Sending DeleteTablet for 3 replicas of tablet 23a9f75a1f3a48588fc0ab9163f97125
I20260619 02:49:57.121508 26902 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a9f75a1f3a48588fc0ab9163f97125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:57 UTC) from {username='slave'} at 127.0.0.1:43462
I20260619 02:49:57.121542 27164 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a9f75a1f3a48588fc0ab9163f97125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:57 UTC) from {username='slave'} at 127.0.0.1:44794
I20260619 02:49:57.121745 27267 tablet_replica.cc:330] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8: stopping tablet replica
I20260619 02:49:57.121768 27033 tablet_service.cc:1514] Processing DeleteTablet for tablet 23a9f75a1f3a48588fc0ab9163f97125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:49:57 UTC) from {username='slave'} at 127.0.0.1:56326
I20260619 02:49:57.121901 27267 raft_consensus.cc:2229] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:57.122006 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 26840
I20260619 02:49:57.122030 27267 raft_consensus.cc:2258] T 23a9f75a1f3a48588fc0ab9163f97125 P 568ab555cbb6480b8cfdd9b0316785e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:57.122049 27269 tablet_replica.cc:330] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: stopping tablet replica
I20260619 02:49:57.122129 27269 raft_consensus.cc:2229] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:49:57.122225 27269 raft_consensus.cc:2258] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:57.122605 27268 tablet_replica.cc:330] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92: stopping tablet replica
I20260619 02:49:57.122671 27269 ts_tablet_manager.cc:1905] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:57.122737 27268 raft_consensus.cc:2229] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:49:57.122856 27268 raft_consensus.cc:2258] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:49:57.123466 27268 ts_tablet_manager.cc:1905] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:49:57.124277 27269 ts_tablet_manager.cc:1918] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:57.124361 27269 log.cc:1196] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/23a9f75a1f3a48588fc0ab9163f97125
I20260619 02:49:57.124614 27269 ts_tablet_manager.cc:1939] T 23a9f75a1f3a48588fc0ab9163f97125 P 633b9d0d127e47e591062c2523964a85: Deleting consensus metadata
I20260619 02:49:57.125012 26767 catalog_manager.cc:4841] TS 633b9d0d127e47e591062c2523964a85 (127.21.164.194:35669): tablet 23a9f75a1f3a48588fc0ab9163f97125 (table test-table [id=e86b6052230e4ee3bdc0be091617023e]) successfully deleted
I20260619 02:49:57.125288 27268 ts_tablet_manager.cc:1918] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:49:57.125430 27268 log.cc:1196] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/23a9f75a1f3a48588fc0ab9163f97125
I20260619 02:49:57.125741 27268 ts_tablet_manager.cc:1939] T 23a9f75a1f3a48588fc0ab9163f97125 P 547346da43564942a69666faed3c9d92: Deleting consensus metadata
I20260619 02:49:57.126194 26767 catalog_manager.cc:4841] TS 547346da43564942a69666faed3c9d92 (127.21.164.195:42639): tablet 23a9f75a1f3a48588fc0ab9163f97125 (table test-table [id=e86b6052230e4ee3bdc0be091617023e]) successfully deleted
W20260619 02:49:57.127508 26766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:36177 (error 108)
I20260619 02:49:57.127748 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 26971
I20260619 02:49:57.128258 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.127911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:36177 (local address 127.0.0.1:43466)
0619 02:49:57.128179 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:57.128223 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
W20260619 02:49:57.128547 26766 catalog_manager.cc:4584] TS 568ab555cbb6480b8cfdd9b0316785e8 (127.21.164.193:36177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23a9f75a1f3a48588fc0ab9163f97125: Network error: Client connection negotiation failed: client connection to 127.21.164.193:36177: connect: Connection refused (error 111)
I20260619 02:49:57.131985 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 27102
I20260619 02:49:57.135903 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 26749
2026-06-19T02:49:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (717 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 02:49:57.154515 22163 mini_oidc.cc:152] Starting JWKS server
I20260619 02:49:57.155555 22163 webserver.cc:460] Webserver started at https://127.0.0.1:42697/ using document root <none> and password file <none>
I20260619 02:49:57.155642 22163 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:49:57.155802 22163 webserver.cc:460] Webserver started at http://127.0.0.1:33243/ using document root <none> and password file <none>
2026-06-19T02:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:57Z Disabled control of system clock
I20260619 02:49:57.163420 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45245
--webserver_interface=127.21.164.254
--webserver_port=0
--builtin_ntp_servers=127.21.164.212:45603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:45245
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42697/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/testchainca.cert with env {}
W20260619 02:49:57.253245 27276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:57.253461 27276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:57.253491 27276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:57.254671 27276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:57.254709 27276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:57.254726 27276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:57.254741 27276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:57.254756 27276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:49:57.254781 27276 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:49:57.254801 27276 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42697/jwks.json
I20260619 02:49:57.256356 27276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:45245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42697/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:57.256597 27276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:57.256817 27276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:57.261307 27284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:57.261313 27282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:57.261502 27276 server_base.cc:1034] running on GCE node
W20260619 02:49:57.261514 27281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:57.261806 27276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:57.262081 27276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:57.263227 27276 hybrid_clock.cc:648] HybridClock initialized: now 1781837397263193 us; error 52 us; skew 500 ppm
I20260619 02:49:57.264458 27276 webserver.cc:460] Webserver started at http://127.21.164.254:42629/ using document root <none> and password file <none>
I20260619 02:49:57.264688 27276 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:57.264740 27276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:57.264834 27276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:57.265851 27276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "09629d77b0104363a461d372d4a7f833"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "b6fff9026e6e95105c9232b274417360"
server_key_iv: "9ccf832f8f7ac777d34c522da43052b5"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.266156 27276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "09629d77b0104363a461d372d4a7f833"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "b6fff9026e6e95105c9232b274417360"
server_key_iv: "9ccf832f8f7ac777d34c522da43052b5"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.267294 27276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:57.267968 27290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:57.268121 27276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:57.268203 27276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "09629d77b0104363a461d372d4a7f833"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "b6fff9026e6e95105c9232b274417360"
server_key_iv: "9ccf832f8f7ac777d34c522da43052b5"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.268260 27276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:57.280298 27276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:57.280602 27276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:57.280784 27276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:57.284507 27276 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45245
I20260619 02:49:57.284545 27342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45245 every 8 connection(s)
I20260619 02:49:57.284945 27276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:57.285861 27343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:57.287503 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 27276
I20260619 02:49:57.287609 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:57.287788 22163 external_mini_cluster.cc:1428] Setting key 9cd5d3284444bf3a76b818985e6b594a
I20260619 02:49:57.288470 27343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833: Bootstrap starting.
I20260619 02:49:57.289101 27343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:57.289369 27343 log.cc:824] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:57.290000 27343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833: No bootstrap required, opened a new log
I20260619 02:49:57.291720 27343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09629d77b0104363a461d372d4a7f833" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45245 } }
I20260619 02:49:57.291857 27343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:57.291888 27343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09629d77b0104363a461d372d4a7f833, State: Initialized, Role: FOLLOWER
I20260619 02:49:57.292040 27343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09629d77b0104363a461d372d4a7f833" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45245 } }
I20260619 02:49:57.292119 27343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:57.292157 27343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:57.292196 27343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:57.292732 27343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09629d77b0104363a461d372d4a7f833" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45245 } }
I20260619 02:49:57.292827 27343 leader_election.cc:302] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09629d77b0104363a461d372d4a7f833; no voters:
I20260619 02:49:57.292990 27343 leader_election.cc:288] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:57.293028 27348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:57.293051 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.288289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39936 (local address 127.21.164.254:45245)
0619 02:49:57.288480 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.288490 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.288550 (+ 60us) server_negotiation.cc:407] Connection header received
0619 02:49:57.289072 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.289079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.289146 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.289289 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.289502 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.290419 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.291165 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.291580 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.291692 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.292393 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:57.292404 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:57.292404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:57.292575 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:57.292608 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:57.292619 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:57.292692 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:57.292766 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 02:49:57.292845 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260619 02:49:57.293120 27348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [term 1 LEADER]: Becoming Leader. State: Replica: 09629d77b0104363a461d372d4a7f833, State: Running, Role: LEADER
I20260619 02:49:57.293200 27348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09629d77b0104363a461d372d4a7f833" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45245 } }
I20260619 02:49:57.293238 27343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:57.293496 27348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09629d77b0104363a461d372d4a7f833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09629d77b0104363a461d372d4a7f833" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45245 } } }
I20260619 02:49:57.293560 27348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:57.293673 27348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09629d77b0104363a461d372d4a7f833. Latest consensus state: current_term: 1 leader_uuid: "09629d77b0104363a461d372d4a7f833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09629d77b0104363a461d372d4a7f833" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45245 } } }
I20260619 02:49:57.293726 27348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:57.293850 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:57.293874 27352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:57.294862 27352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:57.296316 27352 catalog_manager.cc:1347] Generated new cluster ID: 1764b47176aa41f68be5ca1291a45492
I20260619 02:49:57.296362 27352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:57.310029 27352 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:57.310556 27352 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:57.314703 27352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09629d77b0104363a461d372d4a7f833: Generated new TSK 0
I20260619 02:49:57.314888 27352 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:49:57.351100 27373 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 02:49:57.351282 27373 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:49:57.351336 27346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.328189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.21.164.254:45245)
0619 02:49:57.328236 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.328238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.328414 (+ 176us) server_negotiation.cc:407] Connection header received
0619 02:49:57.328524 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.328525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.328538 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.328596 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:49:57.328765 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.329281 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.330045 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.330258 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.330313 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.330508 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:49:57.351084 (+ 20576us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:49:57.351258 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39948: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 02:49:57.351428 27346 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39948: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 02:49:57.351725 27372 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:57.328098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45245 (local address 127.0.0.1:39948)
0619 02:49:57.328342 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:57.328352 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:57.328427 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:57.328612 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:57.328613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:57.328616 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:49:57.328700 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.328703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.329357 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.329358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.329958 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.329960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.330337 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.330339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.330389 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.330408 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:49:57.351251 (+ 20843us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:49:57.351432 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.164.254:45245: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.164.254:45245: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 02:49:57.352147 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 27276
2026-06-19T02:49:57Z chronyd exiting
I20260619 02:49:57.387302 22163 test_util.cc:175] -----------------------------------------------
I20260619 02:49:57.387396 22163 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837375929108-22163-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 02:49:57.388176 22163 mini_oidc.cc:152] Starting JWKS server
I20260619 02:49:57.388892 22163 webserver.cc:460] Webserver started at https://127.0.0.1:44463/ using document root <none> and password file <none>
I20260619 02:49:57.388939 22163 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:49:57.389074 22163 webserver.cc:460] Webserver started at http://127.0.0.1:35899/ using document root <none> and password file <none>
2026-06-19T02:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:57Z Disabled control of system clock
I20260619 02:49:57.396477 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34647
--webserver_interface=127.21.164.254
--webserver_port=0
--builtin_ntp_servers=127.21.164.212:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:34647
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44463/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/testchainca.cert with env {}
W20260619 02:49:57.474934 27379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:57.475152 27379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:57.475174 27379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:57.476387 27379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:57.476436 27379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:57.476447 27379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:57.476459 27379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:57.476470 27379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:49:57.476481 27379 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:49:57.476490 27379 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44463/jwks.json
I20260619 02:49:57.478155 27379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:34647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44463/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:57.478417 27379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:57.478627 27379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:57.483322 27387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:57.483341 27384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:57.483314 27385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:57.483366 27379 server_base.cc:1034] running on GCE node
I20260619 02:49:57.483744 27379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:57.484050 27379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:57.485190 27379 hybrid_clock.cc:648] HybridClock initialized: now 1781837397485172 us; error 34 us; skew 500 ppm
I20260619 02:49:57.486426 27379 webserver.cc:460] Webserver started at http://127.21.164.254:38701/ using document root <none> and password file <none>
I20260619 02:49:57.486653 27379 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:57.486702 27379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:57.486819 27379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:57.487871 27379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "d300bc0ecad547079fbb85fbe21d2b07"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "94ac6e823b710773e9765d097838268f"
server_key_iv: "7cbc4f902992bc4e66cef03a12720450"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.488200 27379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "d300bc0ecad547079fbb85fbe21d2b07"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "94ac6e823b710773e9765d097838268f"
server_key_iv: "7cbc4f902992bc4e66cef03a12720450"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.489511 27379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:49:57.490289 27393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:57.490468 27379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:57.490549 27379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "d300bc0ecad547079fbb85fbe21d2b07"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "94ac6e823b710773e9765d097838268f"
server_key_iv: "7cbc4f902992bc4e66cef03a12720450"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.490604 27379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:57.507521 27379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:57.507813 27379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:57.507925 27379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:57.511574 27379 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:34647
I20260619 02:49:57.511624 27445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:34647 every 8 connection(s)
I20260619 02:49:57.511936 27379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:57.512684 27446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:57.515407 27446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07: Bootstrap starting.
I20260619 02:49:57.515960 27446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:57.516208 27446 log.cc:824] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:57.516896 27446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07: No bootstrap required, opened a new log
I20260619 02:49:57.518476 27446 raft_consensus.cc:348] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d300bc0ecad547079fbb85fbe21d2b07" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34647 } }
I20260619 02:49:57.518615 27446 raft_consensus.cc:374] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:57.518646 27446 raft_consensus.cc:729] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d300bc0ecad547079fbb85fbe21d2b07, State: Initialized, Role: FOLLOWER
I20260619 02:49:57.518786 27446 consensus_queue.cc:260] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d300bc0ecad547079fbb85fbe21d2b07" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34647 } }
I20260619 02:49:57.518862 27446 raft_consensus.cc:388] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:57.518904 27446 raft_consensus.cc:482] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:57.518951 27446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:57.519475 27446 raft_consensus.cc:504] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d300bc0ecad547079fbb85fbe21d2b07" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34647 } }
I20260619 02:49:57.519570 27446 leader_election.cc:302] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d300bc0ecad547079fbb85fbe21d2b07; no voters:
I20260619 02:49:57.519735 27446 leader_election.cc:288] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:57.519786 27449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:57.519958 27449 raft_consensus.cc:686] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [term 1 LEADER]: Becoming Leader. State: Replica: d300bc0ecad547079fbb85fbe21d2b07, State: Running, Role: LEADER
I20260619 02:49:57.519979 27446 sys_catalog.cc:564] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:57.520056 27449 consensus_queue.cc:237] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d300bc0ecad547079fbb85fbe21d2b07" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34647 } }
I20260619 02:49:57.520581 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 27379
I20260619 02:49:57.520717 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:57.520706 27451 sys_catalog.cc:455] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d300bc0ecad547079fbb85fbe21d2b07. Latest consensus state: current_term: 1 leader_uuid: "d300bc0ecad547079fbb85fbe21d2b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d300bc0ecad547079fbb85fbe21d2b07" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34647 } } }
I20260619 02:49:57.520812 27451 sys_catalog.cc:458] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:57.520857 27450 sys_catalog.cc:455] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d300bc0ecad547079fbb85fbe21d2b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d300bc0ecad547079fbb85fbe21d2b07" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34647 } } }
I20260619 02:49:57.520907 27450 sys_catalog.cc:458] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:57.520921 22163 external_mini_cluster.cc:1428] Setting key be8644a8115b2d59c35c772352120ca5
I20260619 02:49:57.521580 27458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:57.522060 27458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:57.523633 27458 catalog_manager.cc:1347] Generated new cluster ID: 5fa6b8aaf7db4aa5a446c16ec1ef4b4a
I20260619 02:49:57.523686 27458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:57.526712 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.521461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.21.164.254:34647)
0619 02:49:57.521674 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.521683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.521714 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:49:57.521850 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.521859 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.521932 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.522085 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.522291 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.523491 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.524604 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.525242 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.525346 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.526049 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:57.526052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:57.526053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:57.526245 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:57.526287 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:57.526301 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:57.526356 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:57.526442 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 02:49:57.526514 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260619 02:49:57.527575 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:57.539045 27458 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:57.539754 27458 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:57.545439 27458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d300bc0ecad547079fbb85fbe21d2b07: Generated new TSK 0
I20260619 02:49:57.545668 27458 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:49:57.571664 27476 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 02:49:57.571743 27476 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:49:57.572142 27467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.557666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36442 (local address 127.21.164.254:34647)
0619 02:49:57.557724 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.557726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.557862 (+ 136us) server_negotiation.cc:407] Connection header received
0619 02:49:57.557921 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.557922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.557935 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.558019 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:49:57.558113 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.558917 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.559684 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.560008 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.560075 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.560192 (+ 117us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:49:57.571892 (+ 11700us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:49:57.572055 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36442: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 02:49:57.572185 27475 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:57.557632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34647 (local address 127.0.0.1:36442)
0619 02:49:57.557791 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:57.557802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:57.557865 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:57.558020 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:57.558020 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:57.558022 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:49:57.558075 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.558076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.559004 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.559005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.559612 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.559615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.560074 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.560075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.560110 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.560114 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:49:57.572012 (+ 11898us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:49:57.572116 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.164.254:34647: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":128,"threads_started":1}
W20260619 02:49:57.572206 27467 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36442: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 02:49:57.572383 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 27379
2026-06-19T02:49:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 02:49:57.609149 22163 mini_oidc.cc:152] Starting JWKS server
I20260619 02:49:57.609834 22163 webserver.cc:460] Webserver started at https://127.0.0.1:39191/ using document root <none> and password file <none>
I20260619 02:49:57.609879 22163 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:49:57.610018 22163 webserver.cc:460] Webserver started at http://127.0.0.1:35671/ using document root <none> and password file <none>
2026-06-19T02:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:57Z Disabled control of system clock
I20260619 02:49:57.617295 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:39251
--webserver_interface=127.21.164.254
--webserver_port=0
--builtin_ntp_servers=127.21.164.212:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:39251
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39191/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:49:57.696699 27482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:57.696914 27482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:57.696944 27482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:57.698104 27482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:57.698141 27482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:57.698153 27482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:57.698164 27482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:57.698174 27482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:49:57.698185 27482 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:49:57.698195 27482 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39191/jwks.json
I20260619 02:49:57.699527 27482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:39251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:39251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39191/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:57.699733 27482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:57.699988 27482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:57.704607 27490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:57.704700 27488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:57.704727 27482 server_base.cc:1034] running on GCE node
W20260619 02:49:57.704703 27487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:57.705080 27482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:57.705404 27482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:57.706555 27482 hybrid_clock.cc:648] HybridClock initialized: now 1781837397706540 us; error 28 us; skew 500 ppm
I20260619 02:49:57.707765 27482 webserver.cc:460] Webserver started at http://127.21.164.254:33657/ using document root <none> and password file <none>
I20260619 02:49:57.707989 27482 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:57.708037 27482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:57.708163 27482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:57.709254 27482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "412fcf8a92464f65b8a6f9f9014468f1"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "1e2f3ee1c4f40eb362a5b37da19ca87f"
server_key_iv: "3ea6c97034345160035efb2cb348ba88"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.709631 27482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "412fcf8a92464f65b8a6f9f9014468f1"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "1e2f3ee1c4f40eb362a5b37da19ca87f"
server_key_iv: "3ea6c97034345160035efb2cb348ba88"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.710952 27482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 02:49:57.711726 27496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:57.711894 27482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:57.711980 27482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "412fcf8a92464f65b8a6f9f9014468f1"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "1e2f3ee1c4f40eb362a5b37da19ca87f"
server_key_iv: "3ea6c97034345160035efb2cb348ba88"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.712045 27482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:57.726042 27482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:57.726383 27482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:57.726523 27482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:57.730190 27482 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:39251
I20260619 02:49:57.730232 27548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:39251 every 8 connection(s)
I20260619 02:49:57.730571 27482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:57.731267 27549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:57.731387 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 27482
I20260619 02:49:57.731480 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:57.731647 22163 external_mini_cluster.cc:1428] Setting key 340514cbeede2499488f99578bb68255
I20260619 02:49:57.735076 27549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1: Bootstrap starting.
I20260619 02:49:57.735723 27549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:57.736012 27549 log.cc:824] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:57.736811 27549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1: No bootstrap required, opened a new log
I20260619 02:49:57.737419 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.732164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38084 (local address 127.21.164.254:39251)
0619 02:49:57.732385 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.732395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.732427 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:49:57.732925 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.732935 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.733020 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.733172 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.733328 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.734616 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.735393 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.735990 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.736104 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.736751 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:57.736755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:57.736756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:57.736956 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:57.736995 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:57.737010 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:57.737074 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:57.737128 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 02:49:57.737204 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260619 02:49:57.738520 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:57.738909 27549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412fcf8a92464f65b8a6f9f9014468f1" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39251 } }
I20260619 02:49:57.739077 27549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:57.739114 27549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412fcf8a92464f65b8a6f9f9014468f1, State: Initialized, Role: FOLLOWER
I20260619 02:49:57.739231 27549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412fcf8a92464f65b8a6f9f9014468f1" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39251 } }
I20260619 02:49:57.739310 27549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:57.739348 27549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:57.739393 27549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:57.739910 27549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412fcf8a92464f65b8a6f9f9014468f1" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39251 } }
I20260619 02:49:57.739992 27549 leader_election.cc:302] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 412fcf8a92464f65b8a6f9f9014468f1; no voters:
I20260619 02:49:57.740104 27549 leader_election.cc:288] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:57.740151 27554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:57.740315 27549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:57.740357 27554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [term 1 LEADER]: Becoming Leader. State: Replica: 412fcf8a92464f65b8a6f9f9014468f1, State: Running, Role: LEADER
I20260619 02:49:57.740455 27554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412fcf8a92464f65b8a6f9f9014468f1" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39251 } }
I20260619 02:49:57.740851 27556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412fcf8a92464f65b8a6f9f9014468f1. Latest consensus state: current_term: 1 leader_uuid: "412fcf8a92464f65b8a6f9f9014468f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412fcf8a92464f65b8a6f9f9014468f1" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39251 } } }
I20260619 02:49:57.740953 27556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:57.740845 27555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "412fcf8a92464f65b8a6f9f9014468f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412fcf8a92464f65b8a6f9f9014468f1" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39251 } } }
I20260619 02:49:57.741065 27555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:57.741202 27563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:57.741793 27563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:57.743211 27563 catalog_manager.cc:1347] Generated new cluster ID: ead86aedf06640ceb944ba0beaffb31a
I20260619 02:49:57.743263 27563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:57.782006 27563 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:57.782574 27563 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:57.794054 27563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 412fcf8a92464f65b8a6f9f9014468f1: Generated new TSK 0
I20260619 02:49:57.794256 27563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:57.808424 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.804964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.21.164.254:39251)
0619 02:49:57.805023 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.805025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.805131 (+ 106us) server_negotiation.cc:407] Connection header received
0619 02:49:57.805216 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.805217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.805227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.805309 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:49:57.805444 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.805911 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.807014 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.807191 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.807233 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.807408 (+ 175us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:49:57.808195 (+ 787us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:49:57.808353 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 02:49:57.808492 27552 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 02:49:57.808493 27578 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:57.804898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39251 (local address 127.0.0.1:38086)
0619 02:49:57.805059 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:57.805071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:57.805146 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:57.805305 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:57.805306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:57.805309 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:49:57.805381 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.805384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.806001 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.806002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.806942 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.806945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.807258 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.807259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.807316 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.807322 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:49:57.808291 (+ 969us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:49:57.808397 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.164.254:39251: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39191/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260619 02:49:57.808974 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 27482
E20260619 02:49:57.821056 27579 webserver.cc:533] Webserver: sslize failed:
E20260619 02:49:57.821141 27579 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T02:49:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T02:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:57Z Disabled control of system clock
I20260619 02:49:57.853931 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:46281
--webserver_interface=127.21.164.254
--webserver_port=0
--builtin_ntp_servers=127.21.164.212:35113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:46281
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/testchainca.cert with env {}
W20260619 02:49:57.932127 27583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:57.932353 27583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:57.932384 27583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:57.933583 27583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:57.933625 27583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:57.933641 27583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:57.933657 27583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:57.933671 27583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:49:57.933688 27583 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:49:57.933707 27583 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 02:49:57.935158 27583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:35113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:46281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:46281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:57.935405 27583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:57.935602 27583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:57.940102 27588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:57.940093 27589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:57.940178 27583 server_base.cc:1034] running on GCE node
W20260619 02:49:57.940095 27591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:57.940613 27583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:57.940907 27583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:57.942068 27583 hybrid_clock.cc:648] HybridClock initialized: now 1781837397942055 us; error 29 us; skew 500 ppm
I20260619 02:49:57.943307 27583 webserver.cc:460] Webserver started at http://127.21.164.254:40229/ using document root <none> and password file <none>
I20260619 02:49:57.943517 27583 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:57.943562 27583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:57.943665 27583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:57.944756 27583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "d35f25335a30401d8e3854d1e0a64512"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "8cc02bcb92ac65c4072447da618d62af"
server_key_iv: "d85e697c5b63750503703d8f3665ba1b"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.945092 27583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "d35f25335a30401d8e3854d1e0a64512"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "8cc02bcb92ac65c4072447da618d62af"
server_key_iv: "d85e697c5b63750503703d8f3665ba1b"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.946367 27583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:49:57.947057 27597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:57.947204 27583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:57.947279 27583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "d35f25335a30401d8e3854d1e0a64512"
format_stamp: "Formatted at 2026-06-19 02:49:57 on dist-test-slave-jt0t"
server_key: "8cc02bcb92ac65c4072447da618d62af"
server_key_iv: "d85e697c5b63750503703d8f3665ba1b"
server_key_version: "encryptionkey@0"
I20260619 02:49:57.947355 27583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:57.962910 27583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:57.963238 27583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:57.963413 27583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:57.967068 27583 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:46281
I20260619 02:49:57.967099 27649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:46281 every 8 connection(s)
I20260619 02:49:57.967423 27583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:57.967856 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 27583
I20260619 02:49:57.967933 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:57.968109 22163 external_mini_cluster.cc:1428] Setting key a6ea01e1b8864fee2d0e6df04ba74885
I20260619 02:49:57.968528 27650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:57.971704 27650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512: Bootstrap starting.
I20260619 02:49:57.972390 27650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:57.972720 27650 log.cc:824] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:57.973505 27650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512: No bootstrap required, opened a new log
I20260619 02:49:57.973968 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.968580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.21.164.254:46281)
0619 02:49:57.968804 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.968813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.968845 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:49:57.969316 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.969325 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.969411 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.969562 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:57.969743 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.971063 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.971873 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.972459 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.972586 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.973244 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:57.973248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:57.973249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:49:57.973444 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:57.973486 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:49:57.973500 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:57.973566 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:57.973682 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 02:49:57.973769 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
I20260619 02:49:57.974869 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:49:57.975659 27650 raft_consensus.cc:348] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f25335a30401d8e3854d1e0a64512" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46281 } }
I20260619 02:49:57.975831 27650 raft_consensus.cc:374] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:57.975874 27650 raft_consensus.cc:729] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35f25335a30401d8e3854d1e0a64512, State: Initialized, Role: FOLLOWER
I20260619 02:49:57.975973 27650 consensus_queue.cc:260] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f25335a30401d8e3854d1e0a64512" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46281 } }
I20260619 02:49:57.976044 27650 raft_consensus.cc:388] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:57.976069 27650 raft_consensus.cc:482] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:57.976101 27650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:57.976599 27650 raft_consensus.cc:504] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f25335a30401d8e3854d1e0a64512" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46281 } }
I20260619 02:49:57.976713 27650 leader_election.cc:302] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d35f25335a30401d8e3854d1e0a64512; no voters:
I20260619 02:49:57.976871 27650 leader_election.cc:288] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:57.976910 27655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:57.977047 27655 raft_consensus.cc:686] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [term 1 LEADER]: Becoming Leader. State: Replica: d35f25335a30401d8e3854d1e0a64512, State: Running, Role: LEADER
I20260619 02:49:57.977115 27650 sys_catalog.cc:564] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:57.977133 27655 consensus_queue.cc:237] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f25335a30401d8e3854d1e0a64512" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46281 } }
I20260619 02:49:57.977494 27657 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d35f25335a30401d8e3854d1e0a64512. Latest consensus state: current_term: 1 leader_uuid: "d35f25335a30401d8e3854d1e0a64512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f25335a30401d8e3854d1e0a64512" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46281 } } }
I20260619 02:49:57.977501 27656 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d35f25335a30401d8e3854d1e0a64512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f25335a30401d8e3854d1e0a64512" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46281 } } }
I20260619 02:49:57.977592 27657 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:57.977596 27656 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512 [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:57.977803 27662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:57.978291 27662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:57.979844 27662 catalog_manager.cc:1347] Generated new cluster ID: 2c4313344db24952834f854dadaefa9e
I20260619 02:49:57.979889 27662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:49:57.984548 27662 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:57.985080 27662 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:57.987269 27662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d35f25335a30401d8e3854d1e0a64512: Generated new TSK 0
I20260619 02:49:57.987432 27662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:49:58.065069 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:57.993438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.21.164.254:46281)
0619 02:49:57.993497 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:57.993499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:57.993569 (+ 70us) server_negotiation.cc:407] Connection header received
0619 02:49:57.993654 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:57.993655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:57.993666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:57.993726 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:49:57.993852 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.994356 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.995128 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.995340 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.995394 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.995556 (+ 162us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:49:58.064777 (+ 69221us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:49:58.064984 (+ 207us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53530: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 02:49:58.065135 27653 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53530: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 02:49:58.065491 27679 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:57.993376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:46281 (local address 127.0.0.1:53530)
0619 02:49:57.993519 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:57.993529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:57.993587 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:57.993729 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:57.993730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:57.993732 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:49:57.993788 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.993790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.994443 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.994444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.995050 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:57.995052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:57.995406 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:57.995407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:57.995464 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:57.995470 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:49:58.065264 (+ 69794us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:49:58.065391 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.164.254:46281: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260619 02:49:58.065833 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 27583
2026-06-19T02:49:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (235 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:49:58 dist-test-slave-jt0t krb5kdc[27691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:49:58 dist-test-slave-jt0t krb5kdc[27691](info): set up 2 sockets
Jun 19 02:49:58 dist-test-slave-jt0t krb5kdc[27691](info): commencing operation
krb5kdc: starting...
W20260619 02:49:59.192632 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:49:59.303606 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:33165
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:35339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:33165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:59.384235 27704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:59.384506 27704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:59.384537 27704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:59.385746 27704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:49:59.385797 27704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:59.385815 27704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:59.385830 27704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:49:59.385845 27704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:49:59.387351 27704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:35339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:33165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:33165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:59.387607 27704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:59.387809 27704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:59.392442 27709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:59.392452 27712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:59.392452 27710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:59.392678 27704 server_base.cc:1034] running on GCE node
I20260619 02:49:59.392930 27704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:59.393172 27704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:59.394309 27704 hybrid_clock.cc:648] HybridClock initialized: now 1781837399394289 us; error 33 us; skew 500 ppm
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.401687 27704 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:49:59.402112 27704 webserver.cc:460] Webserver started at http://127.21.164.254:40531/ using document root <none> and password file <none>
I20260619 02:49:59.402297 27704 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:59.402338 27704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:59.402442 27704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:59.403470 27704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "0e409642cbd04f178a05ff019dd6406a"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "496672536a6f08578dbb22c6888d1b08"
server_key_iv: "dcb31a654ff215342129cff92a1ccc0f"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.403776 27704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "0e409642cbd04f178a05ff019dd6406a"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "496672536a6f08578dbb22c6888d1b08"
server_key_iv: "dcb31a654ff215342129cff92a1ccc0f"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.405037 27704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:49:59.405757 27719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.405951 27704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:49:59.406032 27704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "0e409642cbd04f178a05ff019dd6406a"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "496672536a6f08578dbb22c6888d1b08"
server_key_iv: "dcb31a654ff215342129cff92a1ccc0f"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.406092 27704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:59.424330 27704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:59.426326 27704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:59.426440 27704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:59.430094 27704 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:33165
I20260619 02:49:59.430142 27771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:33165 every 8 connection(s)
I20260619 02:49:59.430454 27704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:49:59.431306 27772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:49:59.433574 27772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a: Bootstrap starting.
I20260619 02:49:59.434122 27772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a: Neither blocks nor log segments found. Creating new log.
I20260619 02:49:59.434384 27772 log.cc:824] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:49:59.435021 27772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a: No bootstrap required, opened a new log
I20260619 02:49:59.436548 27772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e409642cbd04f178a05ff019dd6406a" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33165 } }
I20260619 02:49:59.436718 27772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:49:59.436753 27772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e409642cbd04f178a05ff019dd6406a, State: Initialized, Role: FOLLOWER
I20260619 02:49:59.436901 27772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e409642cbd04f178a05ff019dd6406a" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33165 } }
I20260619 02:49:59.436964 27772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:49:59.436991 27772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:49:59.437038 27772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:49:59.437547 27772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e409642cbd04f178a05ff019dd6406a" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33165 } }
I20260619 02:49:59.437639 27772 leader_election.cc:302] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e409642cbd04f178a05ff019dd6406a; no voters:
I20260619 02:49:59.437808 27772 leader_election.cc:288] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:49:59.437849 27775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:49:59.438047 27772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:49:59.438159 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 27704
I20260619 02:49:59.438233 27775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [term 1 LEADER]: Becoming Leader. State: Replica: 0e409642cbd04f178a05ff019dd6406a, State: Running, Role: LEADER
I20260619 02:49:59.438251 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:49:59.438319 27775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e409642cbd04f178a05ff019dd6406a" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33165 } }
I20260619 02:49:59.438479 22163 external_mini_cluster.cc:1428] Setting key 634c58794045227da79108eca2a73122
I20260619 02:49:59.438688 27776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e409642cbd04f178a05ff019dd6406a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e409642cbd04f178a05ff019dd6406a" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33165 } } }
I20260619 02:49:59.438699 27777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e409642cbd04f178a05ff019dd6406a. Latest consensus state: current_term: 1 leader_uuid: "0e409642cbd04f178a05ff019dd6406a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e409642cbd04f178a05ff019dd6406a" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33165 } } }
I20260619 02:49:59.438776 27776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:59.438788 27777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a [sys.catalog]: This master's current role is: LEADER
I20260619 02:49:59.438967 27784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:49:59.439488 27784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:49:59.441022 27784 catalog_manager.cc:1347] Generated new cluster ID: a34b6e9f953148b28524fe85da3627f2
I20260619 02:49:59.441073 27784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:59.450635 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.439079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.21.164.254:33165)
0619 02:49:59.439723 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.439731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.439765 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:49:59.440107 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.440117 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.440193 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.440352 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.441158 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.442558 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.443589 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.444212 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.444328 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.447046 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:59.447050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:59.447051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:59.447063 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:59.448940 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.449338 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.449341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.449341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.449451 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.449638 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.449638 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.449639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.449923 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:59.450068 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:59.450156 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 02:49:59.450399 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":69,"threads_started":1}
I20260619 02:49:59.473673 27784 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:49:59.474192 27784 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:49:59.480216 27784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e409642cbd04f178a05ff019dd6406a: Generated new TSK 0
I20260619 02:49:59.480407 27784 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:49:59.511144 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:33165
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:35339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:59.591536 27798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:59.591740 27798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:59.591763 27798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:59.592934 27798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:59.592972 27798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:59.592998 27798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:49:59.594435 27798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:35339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:33165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:59.594651 27798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:59.594846 27798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:59.599514 27804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:59.599565 27798 server_base.cc:1034] running on GCE node
W20260619 02:49:59.599514 27806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:59.599550 27803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:59.599910 27798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:59.600149 27798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:59.601325 27798 hybrid_clock.cc:648] HybridClock initialized: now 1781837399601301 us; error 33 us; skew 500 ppm
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.603801 27798 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:49:59.604211 27798 webserver.cc:460] Webserver started at http://127.21.164.193:39463/ using document root <none> and password file <none>
I20260619 02:49:59.604399 27798 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:59.604440 27798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:59.604532 27798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:59.605613 27798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "780474189abf400d8b1ae71c92ab903e"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "4c914134a519089a74d37573324b9acd"
server_key_iv: "e3e54e6f065608d12b1495ff10e64c35"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.605914 27798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "780474189abf400d8b1ae71c92ab903e"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "4c914134a519089a74d37573324b9acd"
server_key_iv: "e3e54e6f065608d12b1495ff10e64c35"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.607116 27798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:49:59.607782 27813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.607946 27798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:59.608028 27798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "780474189abf400d8b1ae71c92ab903e"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "4c914134a519089a74d37573324b9acd"
server_key_iv: "e3e54e6f065608d12b1495ff10e64c35"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.608083 27798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:59.622892 27798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:59.624974 27798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:59.625094 27798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:59.625284 27798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:59.625593 27798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:59.625629 27798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.625653 27798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:59.625669 27798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.630896 27798 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:40543
I20260619 02:49:59.630940 27926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:40543 every 8 connection(s)
I20260619 02:49:59.631266 27798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:49:59.635776 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 27798
I20260619 02:49:59.635872 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:49:59.636042 22163 external_mini_cluster.cc:1428] Setting key 66bb6b1e8f3322b05ef95f591861b0e7
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:59.640841 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.631969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:51731 (local address 127.21.164.254:33165)
0619 02:49:59.632021 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.632024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.632233 (+ 209us) server_negotiation.cc:407] Connection header received
0619 02:49:59.633249 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.633251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.633268 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.633323 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.633996 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.634494 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.635467 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.635674 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.635729 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.637924 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:59.637925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:59.637926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:59.637937 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:59.639354 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.639850 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.639850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.639851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.639916 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.640176 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.640177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.640177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.640276 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:59.640319 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:59.640688 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 02:49:59.640770 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:49:59.641093 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.631938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:33165 (local address 127.21.164.193:51731)
0619 02:49:59.632117 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.632153 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:59.633137 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:59.633405 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:59.633409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:59.633696 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:59.633896 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.633900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.634577 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.634578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.635378 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.635380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.635743 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.635744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.635882 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.636400 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:59.636408 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:59.637803 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:59.639463 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.639471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.639473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.639781 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.639975 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.639976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.639977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.640119 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.640329 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:59.640335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:59.640525 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 02:49:59.640671 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 02:49:59.640848 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260619 02:49:59.641647 27927 heartbeater.cc:347] Connected to a master server at 127.21.164.254:33165
I20260619 02:49:59.641757 27927 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:59.641953 27927 heartbeater.cc:511] Master 127.21.164.254:33165 requested a full tablet report, sending...
I20260619 02:49:59.642436 27736 ts_manager.cc:194] Registered new tserver with Master: 780474189abf400d8b1ae71c92ab903e (127.21.164.193:40543)
I20260619 02:49:59.643033 27736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:51731
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:49:59.659947 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:33165
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:35339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:59.740011 27932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:59.740226 27932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:59.740257 27932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:59.741451 27932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:59.741495 27932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:59.741531 27932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:49:59.742985 27932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:35339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:33165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:59.743221 27932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:59.743434 27932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:59.748337 27940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:59.748356 27937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:59.748342 27938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:59.748570 27932 server_base.cc:1034] running on GCE node
I20260619 02:49:59.748813 27932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:59.749048 27932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:59.750211 27932 hybrid_clock.cc:648] HybridClock initialized: now 1781837399750196 us; error 30 us; skew 500 ppm
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.752684 27932 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:49:59.753114 27932 webserver.cc:460] Webserver started at http://127.21.164.194:39247/ using document root <none> and password file <none>
I20260619 02:49:59.753311 27932 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:59.753358 27932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:59.753508 27932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:59.754573 27932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "5944161ee1f54fe7b202ee743eaf8dc5"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "8a74d943b0562a40372690cff569d513"
server_key_iv: "5d66065aec7535cd94488506cbdf02e6"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.754905 27932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "5944161ee1f54fe7b202ee743eaf8dc5"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "8a74d943b0562a40372690cff569d513"
server_key_iv: "5d66065aec7535cd94488506cbdf02e6"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.756167 27932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 02:49:59.756891 27947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.757032 27932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:59.757112 27932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "5944161ee1f54fe7b202ee743eaf8dc5"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "8a74d943b0562a40372690cff569d513"
server_key_iv: "5d66065aec7535cd94488506cbdf02e6"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.757174 27932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:59.768244 27932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:59.770571 27932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:59.770704 27932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:59.770922 27932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:59.771215 27932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:59.771260 27932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.771337 27932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:59.771363 27932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.776770 27932 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:40483
I20260619 02:49:59.776829 28060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:40483 every 8 connection(s)
I20260619 02:49:59.777141 27932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:59.784595 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 27932
I20260619 02:49:59.784735 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:49:59.784950 22163 external_mini_cluster.cc:1428] Setting key a05ef3699a7c006a1d0cbae5df43ff39
I20260619 02:49:59.786360 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.777831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:56347 (local address 127.21.164.254:33165)
0619 02:49:59.777907 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.777909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.778067 (+ 158us) server_negotiation.cc:407] Connection header received
0619 02:49:59.778976 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.778977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.778988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.779040 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.779621 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.780106 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.781122 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.781343 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.781406 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.783483 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:59.783484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:59.783485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:59.783491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:59.785058 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.785465 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.785465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.785466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.785532 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.785752 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.785752 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.785753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.785889 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:59.785954 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:59.786180 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 02:49:59.786267 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:49:59.786482 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.777812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:33165 (local address 127.21.164.194:56347)
0619 02:49:59.777952 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.777987 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:59.778879 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:59.779100 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:59.779103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:59.779353 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:59.779556 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.779560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.780177 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.780178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.781027 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.781030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.781411 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.781412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.781521 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.782072 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:59.782081 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:59.783377 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:59.785139 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.785147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.785150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.785399 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.785589 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.785590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.785590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.785697 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.785948 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:59.785951 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:59.786071 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 02:49:59.786150 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:49:59.786271 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260619 02:49:59.786911 28061 heartbeater.cc:347] Connected to a master server at 127.21.164.254:33165
I20260619 02:49:59.787024 28061 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:59.787197 28061 heartbeater.cc:511] Master 127.21.164.254:33165 requested a full tablet report, sending...
I20260619 02:49:59.787563 27736 ts_manager.cc:194] Registered new tserver with Master: 5944161ee1f54fe7b202ee743eaf8dc5 (127.21.164.194:40483)
I20260619 02:49:59.788002 27736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:56347
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:49:59.809325 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:33165
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:35339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:49:59.889127 28066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:49:59.889343 28066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:49:59.889375 28066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:49:59.890559 28066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:49:59.890601 28066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:49:59.890635 28066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:49:59.892112 28066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:35339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:33165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:49:59.892346 28066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:49:59.892560 28066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:49:59.897357 28074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:59.897349 28072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:49:59.897349 28071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:49:59.897610 28066 server_base.cc:1034] running on GCE node
I20260619 02:49:59.897830 28066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:49:59.898052 28066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:49:59.899186 28066 hybrid_clock.cc:648] HybridClock initialized: now 1781837399899177 us; error 26 us; skew 500 ppm
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.901667 28066 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:49:59.902127 28066 webserver.cc:460] Webserver started at http://127.21.164.195:39383/ using document root <none> and password file <none>
I20260619 02:49:59.902334 28066 fs_manager.cc:359] Metadata directory not provided
I20260619 02:49:59.902375 28066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:49:59.902505 28066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:49:59.903553 28066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "24263f1f23a148c48eae8aff2cbb1d8b"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "469962deaf8a765e5ce7c9a3fdad1c8d"
server_key_iv: "a4cc44e4d378585b8d06ef5602a47d69"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.903880 28066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "24263f1f23a148c48eae8aff2cbb1d8b"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "469962deaf8a765e5ce7c9a3fdad1c8d"
server_key_iv: "a4cc44e4d378585b8d06ef5602a47d69"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.905166 28066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 02:49:59.905908 28081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.906080 28066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:49:59.906157 28066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "24263f1f23a148c48eae8aff2cbb1d8b"
format_stamp: "Formatted at 2026-06-19 02:49:59 on dist-test-slave-jt0t"
server_key: "469962deaf8a765e5ce7c9a3fdad1c8d"
server_key_iv: "a4cc44e4d378585b8d06ef5602a47d69"
server_key_version: "encryptionkey@0"
I20260619 02:49:59.906210 28066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:49:59.918916 28066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:49:59.921145 28066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:49:59.921257 28066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:49:59.921470 28066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:49:59.921721 28066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:49:59.921751 28066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.921772 28066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:49:59.921785 28066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:49:59.926986 28066 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:33937
I20260619 02:49:59.927029 28194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:33937 every 8 connection(s)
I20260619 02:49:59.927323 28066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:49:59.934024 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 28066
I20260619 02:49:59.934129 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:49:59.934329 22163 external_mini_cluster.cc:1428] Setting key 6cb348f485a05c7476cde389d78736a7
I20260619 02:49:59.936360 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.928032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:58835 (local address 127.21.164.254:33165)
0619 02:49:59.928088 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.928090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.928281 (+ 191us) server_negotiation.cc:407] Connection header received
0619 02:49:59.929274 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.929276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.929290 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.929342 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.929962 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.930458 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.931439 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.931623 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.931676 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.933696 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:59.933698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:59.933698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:59.933706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:59.935085 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.935475 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.935476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.935477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.935537 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.935741 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.935741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.935742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.935861 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:59.935923 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:59.936177 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 02:49:59.936270 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:49:59.936534 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.927998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:33165 (local address 127.21.164.195:58835)
0619 02:49:59.928167 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:49:59.928203 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:49:59.929170 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:49:59.929415 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:49:59.929419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:49:59.929672 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:49:59.929872 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.929877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.930535 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.930536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.931354 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:49:59.931356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.931687 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.931688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:49:59.931805 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.932243 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:49:59.932251 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:49:59.933603 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:49:59.935155 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.935162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.935164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.935408 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.935588 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:49:59.935588 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:49:59.935589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:49:59.935687 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:49:59.935921 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:49:59.935927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:49:59.936057 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 02:49:59.936147 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:49:59.936313 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260619 02:49:59.936925 28195 heartbeater.cc:347] Connected to a master server at 127.21.164.254:33165
I20260619 02:49:59.937021 28195 heartbeater.cc:464] Registering TS with master...
I20260619 02:49:59.937189 28195 heartbeater.cc:511] Master 127.21.164.254:33165 requested a full tablet report, sending...
I20260619 02:49:59.937474 27736 ts_manager.cc:194] Registered new tserver with Master: 24263f1f23a148c48eae8aff2cbb1d8b (127.21.164.195:33937)
I20260619 02:49:59.937892 27736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:58835
I20260619 02:49:59.946035 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/test-admin.keytab.
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.960602 22163 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 02:49:59.962112 28202 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:59.961466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.21.164.193:40543)
0619 02:49:59.961632 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.961637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.961647 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:49:59.961672 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.961674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.961688 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.961749 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.962048 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45856: BlockingRecv error: recv got EOF from 127.0.0.1:45856 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
W20260619 02:49:59.962303 22163 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.963240 22163 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.964069 22163 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:49:59.971138 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.964546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.21.164.193:40543)
0619 02:49:59.964584 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.964586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.964639 (+ 53us) server_negotiation.cc:407] Connection header received
0619 02:49:59.964709 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.964709 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.964719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.964763 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.965089 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.965799 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.966479 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.966780 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.966827 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.968707 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:59.968714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:59.968715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:59.968721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:59.970081 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.970408 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.970411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.970411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.970482 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.970626 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.970627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.970627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.970750 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:59.970861 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:59.971006 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 02:49:59.971067 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 02:49:59.972885 28202 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:59.972311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.21.164.193:40543)
0619 02:49:59.972360 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.972361 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.972410 (+ 49us) server_negotiation.cc:407] Connection header received
0619 02:49:59.972482 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.972482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.972492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.972532 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.972841 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45876: BlockingRecv error: recv got EOF from 127.0.0.1:45876 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 02:49:59.973052 22163 init.cc:244] Error while opening credential cache '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5cc)
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.973966 22163 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.974745 22163 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:49:59.981634 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.975269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.21.164.193:40543)
0619 02:49:59.975338 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.975340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.975428 (+ 88us) server_negotiation.cc:407] Connection header received
0619 02:49:59.975493 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.975494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.975505 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.975548 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.975905 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.976386 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.977091 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.977293 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.977347 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.979220 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:59.979222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:59.979222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:59.979229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:59.980630 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.980964 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.980965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.980966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.981020 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.981175 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.981175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.981176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.981279 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:59.981364 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:59.981458 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 02:49:59.981536 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 02:49:59.982976 28202 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:49:59.982567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.21.164.193:40543)
0619 02:49:59.982605 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.982606 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.982637 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:49:59.982689 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.982690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.982697 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.982737 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.982939 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45906: BlockingRecv error: recv got EOF from 127.0.0.1:45906 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 02:49:59.983052 22163 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837375929108-22163-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.983959 22163 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:49:59.984759 22163 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:49:59 dist-test-slave-jt0t krb5kdc[27691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:49:59.991447 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:49:59.985216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.21.164.193:40543)
0619 02:49:59.985254 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 02:49:59.985255 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:49:59.985285 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:49:59.985351 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:49:59.985352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:49:59.985367 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:49:59.985405 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:49:59.985736 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.986189 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.986839 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:49:59.987019 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:49:59.987071 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:49:59.989011 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:49:59.989012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:49:59.989012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:49:59.989018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:49:59.990382 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.990697 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.990698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.990698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.990748 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:49:59.990923 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:49:59.990924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:49:59.990925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:49:59.991027 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:49:59.991107 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:49:59.991322 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 02:49:59.991384 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 02:49:59.992009 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 27798
I20260619 02:49:59.996215 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 27932
I20260619 02:49:59.999781 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 28066
I20260619 02:50:00.003225 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 27704
2026-06-19T02:50:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1765 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:50:00 dist-test-slave-jt0t krb5kdc[28215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:50:00 dist-test-slave-jt0t krb5kdc[28215](info): set up 2 sockets
Jun 19 02:50:00 dist-test-slave-jt0t krb5kdc[28215](info): commencing operation
krb5kdc: starting...
W20260619 02:50:00.960603 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.254.keytab.
Entry for principal oryx/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.254.keytab.
I20260619 02:50:01.071715 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:41601
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.254.keytab
--principal=oryx/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:41601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:01.151155 28228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:01.151393 28228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:01.151415 28228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:01.152565 28228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:01.152603 28228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:01.152616 28228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:01.152629 28228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:01.152670 28228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:01.154119 28228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:41601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.254.keytab
--principal=oryx/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:41601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:01.154356 28228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:01.154573 28228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:01.159162 28236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:01.159162 28233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:01.159168 28234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:01.159271 28228 server_base.cc:1034] running on GCE node
I20260619 02:50:01.159652 28228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:01.159945 28228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:01.161093 28228 hybrid_clock.cc:648] HybridClock initialized: now 1781837401161075 us; error 34 us; skew 500 ppm
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:01.163689 28228 init.cc:377] Logged in from keytab as oryx/127.21.164.254@KRBTEST.COM (short username oryx)
I20260619 02:50:01.164141 28228 webserver.cc:460] Webserver started at http://127.21.164.254:39999/ using document root <none> and password file <none>
I20260619 02:50:01.164366 28228 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:01.164409 28228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:01.164536 28228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:01.165674 28228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "1c8a057eee6c4376b5c1a06a6154abd0"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "95a36ca259d5767dbc6ecf8b1ea7de33"
server_key_iv: "b0b3f9e34c80ff07728cd29c0658e271"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.165999 28228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "1c8a057eee6c4376b5c1a06a6154abd0"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "95a36ca259d5767dbc6ecf8b1ea7de33"
server_key_iv: "b0b3f9e34c80ff07728cd29c0658e271"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.167227 28228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:50:01.167979 28243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.168140 28228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:50:01.168221 28228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "1c8a057eee6c4376b5c1a06a6154abd0"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "95a36ca259d5767dbc6ecf8b1ea7de33"
server_key_iv: "b0b3f9e34c80ff07728cd29c0658e271"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.168290 28228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:01.184131 28228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:01.185945 28228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:01.186072 28228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:01.189921 28228 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:41601
I20260619 02:50:01.189965 28295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:41601 every 8 connection(s)
I20260619 02:50:01.190320 28228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:50:01.191365 28296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:01.193645 28296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0: Bootstrap starting.
I20260619 02:50:01.194167 28296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:01.194391 28296 log.cc:824] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:01.195024 28296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0: No bootstrap required, opened a new log
I20260619 02:50:01.196404 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 28228
I20260619 02:50:01.196508 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:50:01.196729 22163 external_mini_cluster.cc:1428] Setting key bf89468873ff5c579644e5a1348df419
I20260619 02:50:01.196681 28296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } }
I20260619 02:50:01.196861 28296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:01.196902 28296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8a057eee6c4376b5c1a06a6154abd0, State: Initialized, Role: FOLLOWER
I20260619 02:50:01.197022 28296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } }
I20260619 02:50:01.197090 28296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:50:01.197118 28296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:50:01.197156 28296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:01.197818 28296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } }
I20260619 02:50:01.197928 28296 leader_election.cc:302] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c8a057eee6c4376b5c1a06a6154abd0; no voters:
I20260619 02:50:01.198089 28296 leader_election.cc:288] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:50:01.198160 28301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:01.198299 28301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 1 LEADER]: Becoming Leader. State: Replica: 1c8a057eee6c4376b5c1a06a6154abd0, State: Running, Role: LEADER
I20260619 02:50:01.198360 28296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:01.198392 28301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } }
I20260619 02:50:01.198742 28303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c8a057eee6c4376b5c1a06a6154abd0. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } }
I20260619 02:50:01.198832 28303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:01.198989 28302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } }
I20260619 02:50:01.199048 28302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:01.199147 28308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:01.199636 28308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:50:01.201443 28308 catalog_manager.cc:1347] Generated new cluster ID: c4c10474c1dd4db7a5cc6420f8dc823c
I20260619 02:50:01.201501 28308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:01.208366 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.197193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.21.164.254:41601)
0619 02:50:01.197378 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.197388 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.197421 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:50:01.197810 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.197820 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.197889 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.198040 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.198870 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.200263 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.201309 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.201820 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.201929 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.204483 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.204488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.204488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.204501 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.206884 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.207203 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.207206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.207207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.207325 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.207505 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.207505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.207506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.207757 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.207914 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.207994 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.208133 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260619 02:50:01.218938 28308 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:50:01.219626 28308 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:01.221637 28308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0: Generated new TSK 0
I20260619 02:50:01.221849 28308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab.
Entry for principal oryx/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab.
I20260619 02:50:01.252362 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41601
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab
--principal=oryx/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:01.332077 28322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:01.332290 28322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:01.332312 28322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:01.333501 28322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:01.333547 28322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:01.333575 28322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:50:01.335027 28322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab
--principal=oryx/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:01.335249 28322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:01.335469 28322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:01.340256 28330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:01.340318 28322 server_base.cc:1034] running on GCE node
W20260619 02:50:01.340268 28327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:01.340262 28328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:01.340703 28322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:01.340943 28322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:01.342103 28322 hybrid_clock.cc:648] HybridClock initialized: now 1781837401342087 us; error 28 us; skew 500 ppm
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:01.344540 28322 init.cc:377] Logged in from keytab as oryx/127.21.164.193@KRBTEST.COM (short username oryx)
I20260619 02:50:01.344985 28322 webserver.cc:460] Webserver started at http://127.21.164.193:35737/ using document root <none> and password file <none>
I20260619 02:50:01.345187 28322 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:01.345254 28322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:01.345391 28322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:01.346664 28322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "33bc48e3da76477e9a8136915ad87b63"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "b55e535844a9bfaa7533f52838fb7854"
server_key_iv: "6b2215dac3e622fad354695c2a77e8f3"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.347087 28322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "33bc48e3da76477e9a8136915ad87b63"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "b55e535844a9bfaa7533f52838fb7854"
server_key_iv: "6b2215dac3e622fad354695c2a77e8f3"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.348621 28322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:50:01.349347 28337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.349506 28322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:01.349591 28322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "33bc48e3da76477e9a8136915ad87b63"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "b55e535844a9bfaa7533f52838fb7854"
server_key_iv: "6b2215dac3e622fad354695c2a77e8f3"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.349658 28322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:01.365679 28322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:01.367473 28322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:01.367592 28322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:01.367810 28322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:01.368088 28322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:01.368119 28322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.368142 28322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:01.368165 28322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.373538 28322 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:38869
I20260619 02:50:01.373598 28450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:38869 every 8 connection(s)
I20260619 02:50:01.373885 28322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:50:01.377074 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 28322
I20260619 02:50:01.377169 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:50:01.377339 22163 external_mini_cluster.cc:1428] Setting key 9f7479726e8395805f19df0212d1527e
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:01.382789 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.374560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:36953 (local address 127.21.164.254:41601)
0619 02:50:01.374615 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.374617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.374777 (+ 160us) server_negotiation.cc:407] Connection header received
0619 02:50:01.375280 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.375281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.375329 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.375378 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.376309 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.376854 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.377990 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.378186 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.378239 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.380210 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.380212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.380212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.380219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.381554 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.381918 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.381919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.381919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.381971 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.382183 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.382184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.382184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.382300 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.382356 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.382612 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.382697 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:50:01.382968 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.374525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.193:36953)
0619 02:50:01.374660 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.374701 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.375186 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.375464 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.375469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.376024 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.376209 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.376214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.376954 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.376955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.377928 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.377931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.378249 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.378250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.378375 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.378835 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.378844 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.380124 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:01.381624 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.381632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.381634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.381853 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.382023 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.382024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.382024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.382129 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.382354 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:01.382360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:01.382496 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 02:50:01.382584 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:50:01.382738 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260619 02:50:01.383459 28451 heartbeater.cc:347] Connected to a master server at 127.21.164.254:41601
I20260619 02:50:01.383569 28451 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:01.383759 28451 heartbeater.cc:511] Master 127.21.164.254:41601 requested a full tablet report, sending...
I20260619 02:50:01.384258 28260 ts_manager.cc:194] Registered new tserver with Master: 33bc48e3da76477e9a8136915ad87b63 (127.21.164.193:38869)
WARNING: no policy specified for oryx/127.21.164.194@KRBTEST.COM; defaulting to no policy
I20260619 02:50:01.384955 28260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:36953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab.
Entry for principal oryx/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab.
I20260619 02:50:01.400981 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41601
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab
--principal=oryx/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:01.480116 28456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:01.480334 28456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:01.480357 28456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:01.481526 28456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:01.481559 28456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:01.481585 28456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:50:01.482962 28456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab
--principal=oryx/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:01.483153 28456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:01.483340 28456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:01.488204 28461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:01.488234 28464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:01.488236 28456 server_base.cc:1034] running on GCE node
W20260619 02:50:01.488204 28462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:01.488581 28456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:01.488838 28456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:01.489996 28456 hybrid_clock.cc:648] HybridClock initialized: now 1781837401489980 us; error 29 us; skew 500 ppm
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:01.492509 28456 init.cc:377] Logged in from keytab as oryx/127.21.164.194@KRBTEST.COM (short username oryx)
I20260619 02:50:01.492954 28456 webserver.cc:460] Webserver started at http://127.21.164.194:39051/ using document root <none> and password file <none>
I20260619 02:50:01.493132 28456 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:01.493171 28456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:01.493279 28456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:01.494279 28456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "ca7f70e8e0f340a5b488afc254535a6a"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "87ee628cc8ff81e87155645dc584498d"
server_key_iv: "6d02311eeb40fc21e5106a83f0152077"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.494586 28456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "ca7f70e8e0f340a5b488afc254535a6a"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "87ee628cc8ff81e87155645dc584498d"
server_key_iv: "6d02311eeb40fc21e5106a83f0152077"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.495803 28456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:50:01.496459 28471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.496598 28456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:50:01.496703 28456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "ca7f70e8e0f340a5b488afc254535a6a"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "87ee628cc8ff81e87155645dc584498d"
server_key_iv: "6d02311eeb40fc21e5106a83f0152077"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.496770 28456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:01.516423 28456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:01.518615 28456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:01.518762 28456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:01.518994 28456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:01.519349 28456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:01.519387 28456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.519416 28456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:01.519439 28456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.524842 28456 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:40493
I20260619 02:50:01.524901 28584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:40493 every 8 connection(s)
I20260619 02:50:01.525197 28456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:50:01.525426 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 28456
I20260619 02:50:01.525528 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:50:01.525741 22163 external_mini_cluster.cc:1428] Setting key adc448a6e2d5abc25b7f4e77efae63a7
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.164.195@KRBTEST.COM; defaulting to no policy
I20260619 02:50:01.535168 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.526024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:60773 (local address 127.21.164.254:41601)
0619 02:50:01.526086 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.526088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.526293 (+ 205us) server_negotiation.cc:407] Connection header received
0619 02:50:01.526839 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.526840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.526852 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.526900 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.528044 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.528571 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.530014 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.530214 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.530270 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.532617 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.532619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.532619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.532627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.533916 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.534268 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.534270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.534270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.534325 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.534574 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.534575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.534575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.534693 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.534767 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.535024 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.535098 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:50:01.535276 28587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.526028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.194:60773)
0619 02:50:01.526182 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.526224 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.526733 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.526974 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.526978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.527628 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.527925 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.527932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.528692 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.528694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.529928 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.529931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.530297 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.530298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.530452 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.530987 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.530998 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.532524 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:01.533985 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.533992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.533994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.534204 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.534377 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.534378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.534378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.534505 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.534749 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:01.534753 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:01.534870 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 02:50:01.534962 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:50:01.535085 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260619 02:50:01.535725 28585 heartbeater.cc:347] Connected to a master server at 127.21.164.254:41601
I20260619 02:50:01.535818 28585 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.195@KRBTEST.COM" created.
I20260619 02:50:01.536010 28585 heartbeater.cc:511] Master 127.21.164.254:41601 requested a full tablet report, sending...
I20260619 02:50:01.536392 28260 ts_manager.cc:194] Registered new tserver with Master: ca7f70e8e0f340a5b488afc254535a6a (127.21.164.194:40493)
I20260619 02:50:01.536818 28260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:60773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab.
Entry for principal oryx/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab.
I20260619 02:50:01.551620 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41601
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab
--principal=oryx/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:01.631469 28590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:01.631688 28590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:01.631712 28590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:01.632942 28590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:01.632992 28590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:01.633020 28590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:50:01.634519 28590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab
--principal=oryx/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:01.634783 28590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:01.635013 28590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:01.639891 28596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:01.639909 28595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:01.640010 28590 server_base.cc:1034] running on GCE node
W20260619 02:50:01.639896 28598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:01.640302 28590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:01.640558 28590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:01.641707 28590 hybrid_clock.cc:648] HybridClock initialized: now 1781837401641700 us; error 35 us; skew 500 ppm
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:01.644134 28590 init.cc:377] Logged in from keytab as oryx/127.21.164.195@KRBTEST.COM (short username oryx)
I20260619 02:50:01.644573 28590 webserver.cc:460] Webserver started at http://127.21.164.195:46139/ using document root <none> and password file <none>
I20260619 02:50:01.644819 28590 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:01.644877 28590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:01.645009 28590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:01.646063 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "1c38c65a824845b4b92ab1da928b5f06"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "86dffdb40ccb8cf32e16aa051e2d5680"
server_key_iv: "67c19854e3b0250fffa869c488074e7d"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.646391 28590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "1c38c65a824845b4b92ab1da928b5f06"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "86dffdb40ccb8cf32e16aa051e2d5680"
server_key_iv: "67c19854e3b0250fffa869c488074e7d"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.647652 28590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:50:01.648309 28605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.648468 28590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:01.648550 28590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "1c38c65a824845b4b92ab1da928b5f06"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "86dffdb40ccb8cf32e16aa051e2d5680"
server_key_iv: "67c19854e3b0250fffa869c488074e7d"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.648615 28590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:01.667522 28590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:01.669636 28590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:01.669775 28590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:01.669988 28590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:01.670279 28590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:01.670311 28590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.670341 28590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:01.670365 28590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.676074 28590 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:46603
I20260619 02:50:01.676189 28718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:46603 every 8 connection(s)
I20260619 02:50:01.676460 28590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:01.684522 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.677162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:51571 (local address 127.21.164.254:41601)
0619 02:50:01.677219 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.677221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.677433 (+ 212us) server_negotiation.cc:407] Connection header received
0619 02:50:01.677899 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.677899 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.677908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.677962 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.678836 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.679313 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.680262 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.680471 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.680524 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.682466 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.682468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.682468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.682475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.683522 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.683846 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.683847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.683847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.683892 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.684045 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.684045 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.684045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.684133 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.684177 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.684377 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.684457 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:50:01.684689 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.677140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.195:51571)
0619 02:50:01.677293 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.677334 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.677821 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.678017 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.678020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.678562 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.678753 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.678757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.679383 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.679384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.680188 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.680190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.680538 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.680539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.680676 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.681105 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.681113 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.682384 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:01.683584 (+ 1200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.683590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.683591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.683792 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.683928 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.683928 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.683929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.684009 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.684174 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:01.684177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:01.684284 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 02:50:01.684359 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 02:50:01.684484 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260619 02:50:01.685096 28719 heartbeater.cc:347] Connected to a master server at 127.21.164.254:41601
I20260619 02:50:01.685175 28719 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:01.685341 28719 heartbeater.cc:511] Master 127.21.164.254:41601 requested a full tablet report, sending...
I20260619 02:50:01.685683 28260 ts_manager.cc:194] Registered new tserver with Master: 1c38c65a824845b4b92ab1da928b5f06 (127.21.164.195:46603)
I20260619 02:50:01.686021 28260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:51571
I20260619 02:50:01.686399 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 28590
I20260619 02:50:01.686491 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:50:01.686655 22163 external_mini_cluster.cc:1428] Setting key acf5d79e26e1a6d9043c802f34077caa
I20260619 02:50:01.687372 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.164.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.253.keytab.
Entry for principal oryx/127.21.164.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.253.keytab.
I20260619 02:50:01.710886 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.253:33753
--webserver_interface=127.21.164.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.253.keytab
--principal=oryx/127.21.164.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:41601,127.21.164.253:33753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:01.791438 28724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:01.791656 28724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:01.791683 28724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:01.792896 28724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:01.792939 28724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:01.792958 28724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:01.792973 28724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:01.792989 28724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:01.794499 28724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:41601,127.21.164.253:33753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.253.keytab
--principal=oryx/127.21.164.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.253:33753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.164.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
W20260619 02:50:01.794618 28724 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.164.254:41601,127.21.164.253:33753'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 02:50:01.794764 28724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:01.794957 28724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:01.799708 28732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:01.799769 28729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:01.799849 28724 server_base.cc:1034] running on GCE node
W20260619 02:50:01.799714 28730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:01.800230 28724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:01.800549 28724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:01.801707 28724 hybrid_clock.cc:648] HybridClock initialized: now 1781837401801693 us; error 25 us; skew 500 ppm
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:01.804093 28724 init.cc:377] Logged in from keytab as oryx/127.21.164.253@KRBTEST.COM (short username oryx)
I20260619 02:50:01.804544 28724 webserver.cc:460] Webserver started at http://127.21.164.253:32967/ using document root <none> and password file <none>
I20260619 02:50:01.804766 28724 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:01.804847 28724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:01.804983 28724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:01.806022 28724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data/instance:
uuid: "868b192a035540c49640f0e4caa8e472"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "6b073b2dc5c399c8fb2181b47f2cc273"
server_key_iv: "e08008b4eca32b9244ef3fe3d5e3c005"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.806363 28724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/wal/instance:
uuid: "868b192a035540c49640f0e4caa8e472"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "6b073b2dc5c399c8fb2181b47f2cc273"
server_key_iv: "e08008b4eca32b9244ef3fe3d5e3c005"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.807585 28724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:50:01.808223 28739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.808382 28724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:01.808473 28724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/wal
uuid: "868b192a035540c49640f0e4caa8e472"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "6b073b2dc5c399c8fb2181b47f2cc273"
server_key_iv: "e08008b4eca32b9244ef3fe3d5e3c005"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.808530 28724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:01.822712 28724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:01.825256 28724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:01.825397 28724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:01.829083 28724 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.253:33753
I20260619 02:50:01.829116 28791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.253:33753 every 8 connection(s)
I20260619 02:50:01.829469 28724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
I20260619 02:50:01.830446 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:01.831931 28792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:50:01.835566 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 28724
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.253@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.164.253@KRBTEST.COM
I20260619 02:50:01.841351 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.832715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.21.164.254:41601)
0619 02:50:01.832777 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.832779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.832944 (+ 165us) server_negotiation.cc:407] Connection header received
0619 02:50:01.833435 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.833436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.833446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.833506 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.834110 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.834613 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.835658 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.835913 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.835977 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.838314 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.838316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.838317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.838326 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.839771 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.840303 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.840305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.840305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.840364 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.840601 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.840602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.840602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.840833 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.840908 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.841158 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.841240 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:50:01.841480 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.832663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52114)
0619 02:50:01.832823 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.832866 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.833342 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.833562 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.833565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.833812 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.834008 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.834012 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.834702 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.834703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.835554 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.835557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.836019 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.836020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.836187 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.836734 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.836744 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.838222 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:01.839864 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.839871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.839873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.840187 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.840434 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.840435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.840435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.840542 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.840897 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:01.840901 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:01.841028 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 02:50:01.841127 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:50:01.841261 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260619 02:50:01.842177 28792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:50:01.844209 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.836062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.21.164.253:33753)
0619 02:50:01.836208 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.836213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.836280 (+ 67us) server_negotiation.cc:407] Connection header received
0619 02:50:01.836353 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.836356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.836407 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.836479 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.837219 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.837892 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.838718 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.839015 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.839073 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.841526 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.841533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.841533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.841539 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.843134 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.843460 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.843462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.843463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.843517 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.843676 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.843677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.843678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.843821 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.843995 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.844035 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.844101 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.253@KRBTEST.COM for oryx/127.21.164.253@KRBTEST.COM
I20260619 02:50:01.848764 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.842440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:33753 (local address 127.0.0.1:60224)
0619 02:50:01.842476 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.842485 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.842516 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.842802 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.842803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.842918 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.842958 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.842959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.843551 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.843552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.844032 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.844033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.844435 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.844436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.844525 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.844911 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.844915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.846064 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:01.847888 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.847888 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.847889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.848042 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.848190 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.848190 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.848190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.848234 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.848441 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:01.848442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:01.848483 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 02:50:01.848542 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:50:01.848620 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260619 02:50:01.848824 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.842557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.21.164.253:33753)
0619 02:50:01.842713 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.842715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.842720 (+ 5us) server_negotiation.cc:407] Connection header received
0619 02:50:01.842732 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.842733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.842740 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.842794 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.843016 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.843490 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.844095 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.844368 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.844430 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.846136 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.846137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.846137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.846143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.847808 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.848100 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.848100 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.848100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.848146 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.848274 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.848274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.848274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.848367 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.848410 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.848562 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.848685 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260619 02:50:01.850055 28792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: Bootstrap starting.
I20260619 02:50:01.850644 28792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:01.850889 28792 log.cc:824] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:01.851585 28792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: No bootstrap required, opened a new log
I20260619 02:50:01.853130 28792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } }
I20260619 02:50:01.853258 28792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:01.853289 28792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868b192a035540c49640f0e4caa8e472, State: Initialized, Role: FOLLOWER
I20260619 02:50:01.853405 28792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } }
I20260619 02:50:01.853662 28792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:01.853624 28799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } } }
I20260619 02:50:01.853701 28799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:50:01.855022 28810 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:50:01.855074 28810 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:50:01.856458 28724 master_runner.cc:428] Detected that this master 868b192a035540c49640f0e4caa8e472 is joining an existing cluster
I20260619 02:50:01.856504 28724 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.164.253:33753
I20260619 02:50:01.863790 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.857516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.21.164.254:41601)
0619 02:50:01.857566 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.857568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.857684 (+ 116us) server_negotiation.cc:407] Connection header received
0619 02:50:01.857753 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.857754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.857763 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.857805 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.858141 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.858632 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.859679 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.859869 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.859923 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.860962 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.860964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.860964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.860972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.862610 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.862978 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.862979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.862979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.863042 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.863243 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.863244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.863245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.863381 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.863458 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.863631 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.863713 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:50:01.864081 28816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.857449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52122)
0619 02:50:01.857623 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.857634 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.857696 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.857811 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.857812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.858001 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.858078 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.858080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.858700 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.858701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.859593 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.859596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.859948 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.859949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.860015 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.860428 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.860433 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.860886 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:01.862691 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.862693 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.862693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.862906 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.863105 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.863106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.863106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.863172 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.863447 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:01.863448 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:01.863492 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:50:01.863592 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 02:50:01.863972 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260619 02:50:01.867513 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.857489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:33753 (local address 127.0.0.1:60238)
0619 02:50:01.860766 (+ 3277us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.860776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.860832 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.860982 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.860982 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.861182 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.861244 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.861246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.862069 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.862069 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.863121 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.863123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.863520 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.863521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.863577 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.864085 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.864089 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.864439 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:01.866070 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.866071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.866071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.866615 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.866814 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.866815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.866815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.866872 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.867140 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:01.867140 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:01.867194 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 02:50:01.867259 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:50:01.867366 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3260,"thread_start_us":53,"threads_started":1}
I20260619 02:50:01.867784 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.857599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.21.164.253:33753)
0619 02:50:01.857632 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.857633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.860833 (+ 3200us) server_negotiation.cc:407] Connection header received
0619 02:50:01.860889 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.860889 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.860899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.860948 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.861308 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.861989 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.863179 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.863470 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.863523 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.864513 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.864514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.864514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.864520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.866001 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.866701 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.866702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.866702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.866756 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.866935 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.866936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.866936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.867083 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.867145 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.867289 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.867428 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":15}
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.254@KRBTEST.COM for oryx/127.21.164.253@KRBTEST.COM
I20260619 02:50:01.873428 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.866266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:33753 (local address 127.0.0.1:60252)
0619 02:50:01.866404 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.866443 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.866507 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.866629 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.866632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.866963 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.867026 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.867028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.867910 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.867911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.868721 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.868724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.869067 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.869068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.869151 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.869546 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.869551 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.870766 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:01.872350 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.872357 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.872359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.872556 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.872758 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.872759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.872759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.872824 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.873127 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:01.873132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:01.873176 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:50:01.873240 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:50:01.873335 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260619 02:50:01.873452 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.866315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.21.164.253:33753)
0619 02:50:01.866356 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.866358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.866497 (+ 139us) server_negotiation.cc:407] Connection header received
0619 02:50:01.866571 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.866572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.866581 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.866625 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.867126 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.867816 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.868792 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.869009 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.869061 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.870839 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.870840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.870841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.870847 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.872259 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.872621 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.872622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.872622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.872704 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.872938 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.872938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.872939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.873069 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.873137 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.873274 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.873350 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:50:01.874094 28260 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.164.253:33753: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "868b192a035540c49640f0e4caa8e472"
member_type: NON_VOTER
last_known_addr {
host: "127.21.164.253"
port: 33753
}
attrs {
promote: true
}
}
dest_uuid: "1c8a057eee6c4376b5c1a06a6154abd0"
cas_config_opid_index: -1
I20260619 02:50:01.874389 28260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } }
I20260619 02:50:01.874859 28819 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 868b192a035540c49640f0e4caa8e472 (127.21.164.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } } }
I20260619 02:50:01.875141 28766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:01.875721 28819 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.164.253:33753
I20260619 02:50:01.875800 28819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } } }
I20260619 02:50:01.875862 28819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:01.876053 28766 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Refusing update from remote peer 1c8a057eee6c4376b5c1a06a6154abd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 02:50:01.876217 28724 master.cc:537] Master@127.21.164.253:33753 shutting down...
I20260619 02:50:01.876361 28820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:50:01.877562 28244 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 02:50:01.877980 28244 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 -> Peer 868b192a035540c49640f0e4caa8e472 (127.21.164.253:33753): Couldn't send request to peer 868b192a035540c49640f0e4caa8e472. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:50:01.879017 28724 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:01.879103 28724 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:01.879151 28724 tablet_replica.cc:330] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: stopping tablet replica
I20260619 02:50:01.891963 28724 master.cc:552] Master@127.21.164.253:33753 shutdown complete.
I20260619 02:50:01.892025 28724 master_runner.cc:305] Clearing existing system tablet
I20260619 02:50:01.892148 28724 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:01.892191 28724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:01.892853 28724 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260619 02:50:01.893277 28824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.893450 28724 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:01.893501 28724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.893523 28724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/wal
uuid: "868b192a035540c49640f0e4caa8e472"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "6b073b2dc5c399c8fb2181b47f2cc273"
server_key_iv: "e08008b4eca32b9244ef3fe3d5e3c005"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.894125 28724 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:01.895502 28724 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:50:01.895557 28724 log.cc:1196] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 02:50:01.895722 28724 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: Deleting consensus metadata
I20260619 02:50:01.895817 28724 master_runner.cc:315] Copying system tablet from 127.21.164.254:41601
I20260619 02:50:01.896338 28724 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: tablet copy: Beginning tablet copy session from remote peer at address 127.21.164.254:41601
I20260619 02:50:01.902026 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.896715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.21.164.254:41601)
0619 02:50:01.896760 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.896762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.896900 (+ 138us) server_negotiation.cc:407] Connection header received
0619 02:50:01.896961 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.896961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.896971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.897014 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.897304 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.897796 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.898549 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.898754 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.898806 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.899744 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.899746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.899746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.899753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.901062 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.901353 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.901354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.901355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.901407 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.901564 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.901565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.901565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.901670 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.901726 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.901856 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.901930 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:50:01.902047 28830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.896643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52134)
0619 02:50:01.896818 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:01.896826 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:01.896898 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:01.897027 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:01.897028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:01.897186 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:01.897239 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.897240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.897866 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.897867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.898484 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:01.898486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.898808 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.898809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:01.898866 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.899339 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:01.899345 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:01.899669 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:01.901131 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.901132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.901132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.901294 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.901458 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:01.901459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:01.901459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:01.901516 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:01.901722 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:01.901723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:01.901768 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 02:50:01.901827 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:50:01.901932 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260619 02:50:01.902539 28280 tablet_copy_service.cc:140] P 1c8a057eee6c4376b5c1a06a6154abd0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 868b192a035540c49640f0e4caa8e472 ({username='oryx', principal='oryx/127.21.164.253@KRBTEST.COM'} at 127.0.0.1:52134)
I20260619 02:50:01.902623 28280 tablet_copy_service.cc:161] P 1c8a057eee6c4376b5c1a06a6154abd0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 868b192a035540c49640f0e4caa8e472 at {username='oryx', principal='oryx/127.21.164.253@KRBTEST.COM'} at 127.0.0.1:52134: session id = 868b192a035540c49640f0e4caa8e472-00000000000000000000000000000000
I20260619 02:50:01.903246 28280 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:50:01.903846 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:50:01.904793 28724 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: tablet copy: Starting download of 0 data blocks...
I20260619 02:50:01.904914 28724 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: tablet copy: Starting download of 1 WAL segments...
I20260619 02:50:01.905681 28724 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:50:01.906410 28280 tablet_copy_service.cc:342] P 1c8a057eee6c4376b5c1a06a6154abd0: Request end of tablet copy session 868b192a035540c49640f0e4caa8e472-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.164.253@KRBTEST.COM'} at 127.0.0.1:52134
I20260619 02:50:01.906461 28280 tablet_copy_service.cc:433] P 1c8a057eee6c4376b5c1a06a6154abd0: ending tablet copy session 868b192a035540c49640f0e4caa8e472-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 868b192a035540c49640f0e4caa8e472
I20260619 02:50:01.908512 28724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:01.909780 28832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:01.909832 28831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:01.909811 28834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:01.910033 28724 server_base.cc:1034] running on GCE node
I20260619 02:50:01.910211 28724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:01.910346 28724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:01.911456 28724 hybrid_clock.cc:648] HybridClock initialized: now 1781837401911444 us; error 27 us; skew 500 ppm
Jun 19 02:50:01 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:01.912815 28724 init.cc:377] Logged in from keytab as oryx/127.21.164.253@KRBTEST.COM (short username oryx)
I20260619 02:50:01.913074 28724 webserver.cc:460] Webserver started at http://127.21.164.253:36951/ using document root <none> and password file <none>
I20260619 02:50:01.913185 28724 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:01.913215 28724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:01.913749 28724 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:50:01.914211 28841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.914342 28724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:01.914403 28724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/wal
uuid: "868b192a035540c49640f0e4caa8e472"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "6b073b2dc5c399c8fb2181b47f2cc273"
server_key_iv: "e08008b4eca32b9244ef3fe3d5e3c005"
server_key_version: "encryptionkey@0"
I20260619 02:50:01.914458 28724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:01.922287 28724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:01.923839 28724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:01.926731 28724 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.253:33753
I20260619 02:50:01.927120 28893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.253:33753 every 8 connection(s)
I20260619 02:50:01.927201 28724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
I20260619 02:50:01.927856 28894 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:50:01.928277 28894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: Bootstrap starting.
I20260619 02:50:01.930423 28894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:50:01.930723 28894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: Bootstrap complete.
I20260619 02:50:01.930914 28894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } }
I20260619 02:50:01.930986 28894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 868b192a035540c49640f0e4caa8e472, State: Initialized, Role: LEARNER
I20260619 02:50:01.931011 28894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } }
I20260619 02:50:01.931180 28894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:01.932277 28897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } } }
I20260619 02:50:01.932487 28908 catalog_manager.cc:1259] Loaded cluster ID: c4c10474c1dd4db7a5cc6420f8dc823c
I20260619 02:50:01.932518 28897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:50:01.932724 28908 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: loading cluster ID for follower catalog manager: success
I20260619 02:50:01.933635 28908 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: acquiring CA information for follower catalog manager: success
I20260619 02:50:01.933897 28908 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:50:01.953104 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:01.945835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.21.164.253:33753)
0619 02:50:01.945955 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:01.945957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:01.945966 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:01.945990 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:01.945991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:01.946001 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:01.946068 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:01.946749 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.947446 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.948519 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:01.948776 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:01.948844 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:01.950343 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:01.950344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:01.950344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:01.950351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:01.951856 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.952219 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.952220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.952220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.952281 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:01.952514 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:01.952515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:01.952516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:01.952635 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:01.952756 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:01.952878 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 02:50:01.952961 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for oryx/127.21.164.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.252.keytab.
Entry for principal oryx/127.21.164.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.252.keytab.
I20260619 02:50:01.977596 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.252:37545
--webserver_interface=127.21.164.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.252.keytab
--principal=oryx/127.21.164.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:41601,127.21.164.253:33753,127.21.164.252:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:02.061225 28912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:02.061455 28912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:02.061478 28912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:02.062731 28912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:02.062785 28912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:02.062798 28912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:02.062810 28912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:02.062821 28912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:02.064436 28912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:41601,127.21.164.253:33753,127.21.164.252:37545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.252.keytab
--principal=oryx/127.21.164.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.252:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.164.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:02.064725 28912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:02.065002 28912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:02.069880 28917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:02.070034 28920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:02.069880 28918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:02.070144 28912 server_base.cc:1034] running on GCE node
I20260619 02:50:02.070415 28912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:02.070784 28912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:02.071997 28912 hybrid_clock.cc:648] HybridClock initialized: now 1781837402071976 us; error 40 us; skew 500 ppm
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:02.074527 28912 init.cc:377] Logged in from keytab as oryx/127.21.164.252@KRBTEST.COM (short username oryx)
I20260619 02:50:02.074991 28912 webserver.cc:460] Webserver started at http://127.21.164.252:37459/ using document root <none> and password file <none>
I20260619 02:50:02.075174 28912 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:02.075213 28912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:02.075304 28912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:02.076362 28912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data/instance:
uuid: "71dc4518362c47f2a5b5374fd3693ce6"
format_stamp: "Formatted at 2026-06-19 02:50:02 on dist-test-slave-jt0t"
server_key: "779cf8345aeca3596fa061598c835e8d"
server_key_iv: "2b9fbc39433005ee4874eebca295473e"
server_key_version: "encryptionkey@0"
I20260619 02:50:02.076700 28912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/wal/instance:
uuid: "71dc4518362c47f2a5b5374fd3693ce6"
format_stamp: "Formatted at 2026-06-19 02:50:02 on dist-test-slave-jt0t"
server_key: "779cf8345aeca3596fa061598c835e8d"
server_key_iv: "2b9fbc39433005ee4874eebca295473e"
server_key_version: "encryptionkey@0"
I20260619 02:50:02.077934 28912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:02.078660 28927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.078845 28912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:02.078938 28912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/wal
uuid: "71dc4518362c47f2a5b5374fd3693ce6"
format_stamp: "Formatted at 2026-06-19 02:50:02 on dist-test-slave-jt0t"
server_key: "779cf8345aeca3596fa061598c835e8d"
server_key_iv: "2b9fbc39433005ee4874eebca295473e"
server_key_version: "encryptionkey@0"
I20260619 02:50:02.078996 28912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:02.109658 28912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:02.111845 28912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:02.111976 28912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:02.116192 28912 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.252:37545
I20260619 02:50:02.116242 28979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.252:37545 every 8 connection(s)
I20260619 02:50:02.116720 28912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
I20260619 02:50:02.117899 28980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:02.119436 28980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:50:02.122716 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 28912
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.252@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.164.252@KRBTEST.COM
I20260619 02:50:02.130376 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.120285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.21.164.254:41601)
0619 02:50:02.120342 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.120344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.120461 (+ 117us) server_negotiation.cc:407] Connection header received
0619 02:50:02.120984 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.120985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.121005 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.121073 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.121912 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.122561 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.123974 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.124186 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.124272 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.127129 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.127132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.127132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.127140 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.128804 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.129247 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.129248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.129249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.129348 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.129654 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.129655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.129655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.129774 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.129834 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.130185 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.130266 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:50:02.130586 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.120118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52138)
0619 02:50:02.120297 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.120342 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.120851 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.121177 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.121182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.121560 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.121789 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.121795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.122675 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.122676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.123875 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.123877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.124330 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.124331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.124462 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.124995 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.125004 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.126772 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.128880 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.128889 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.128892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.129154 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.129431 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.129432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.129432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.129587 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.129836 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.129844 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.130015 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.130140 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.130346 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260619 02:50:02.131114 28980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:50:02.132171 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.123228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.21.164.252:37545)
0619 02:50:02.123368 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.123373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.123389 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:50:02.123463 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.123468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.123493 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.123568 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.124217 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.125019 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.125942 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.126416 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.126813 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.129099 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.129105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.129105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.129113 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.130896 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.131238 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.131242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.131242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.131311 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.131563 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.131564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.131564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.131762 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.131925 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.131993 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.132067 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.252@KRBTEST.COM for oryx/127.21.164.253@KRBTEST.COM
I20260619 02:50:02.137867 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.131392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:33753 (local address 127.0.0.1:60270)
0619 02:50:02.131572 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.131582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.131619 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.131739 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.131740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.131844 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.131896 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.131897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.132769 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.132770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.133405 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.133407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.133741 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.133742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.133810 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.134152 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.134156 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.135327 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.136901 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.136902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.136903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.137052 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.137199 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.137200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.137200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.137243 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.137564 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.137565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.137619 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.137698 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.137777 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260619 02:50:02.137923 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.131497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.21.164.253:33753)
0619 02:50:02.131555 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.131557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.131609 (+ 52us) server_negotiation.cc:407] Connection header received
0619 02:50:02.131661 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.131663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.131677 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.131736 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.131967 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.132698 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.133471 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.133685 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.133740 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.135433 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.135434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.135435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.135442 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.136828 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.137110 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.137111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.137111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.137156 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.137278 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.137278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.137278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.137377 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.137438 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.137750 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.137844 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:50:02.138487 28980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.252@KRBTEST.COM for oryx/127.21.164.252@KRBTEST.COM
I20260619 02:50:02.144486 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.138731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:37545 (local address 127.0.0.1:50992)
0619 02:50:02.138772 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.138783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.138828 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.138915 (+ 87us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.138916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.139020 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.139052 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.139053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.139665 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.139666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.140161 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.140163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.140502 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.140503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.140548 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.140909 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.140913 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.142030 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.143553 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.143554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.143555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.143713 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.143931 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.143932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.143933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.143990 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.144185 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.144185 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.144219 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.144261 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.144375 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 02:50:02.144526 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.138790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.21.164.252:37545)
0619 02:50:02.138821 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.138823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.138831 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:02.138866 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.138867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.138874 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.138913 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.139096 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.139586 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.140220 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.140421 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.140467 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.142118 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.142120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.142120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.142128 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.143472 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.143793 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.143794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.143794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.143849 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.144042 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.144042 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.144043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.144143 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.144188 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.144282 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.144426 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":15}
I20260619 02:50:02.145918 28980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: Bootstrap starting.
I20260619 02:50:02.146739 28980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:02.146992 28980 log.cc:824] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:02.147790 28980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: No bootstrap required, opened a new log
I20260619 02:50:02.149744 28980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } }
I20260619 02:50:02.149883 28980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:02.149915 28980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71dc4518362c47f2a5b5374fd3693ce6, State: Initialized, Role: FOLLOWER
I20260619 02:50:02.150029 28980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } }
I20260619 02:50:02.150321 28980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:02.150286 28985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } } }
I20260619 02:50:02.150375 28985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:50:02.152002 28996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:50:02.152055 28996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:50:02.154156 28912 master_runner.cc:428] Detected that this master 71dc4518362c47f2a5b5374fd3693ce6 is joining an existing cluster
I20260619 02:50:02.154207 28912 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.164.252:37545
I20260619 02:50:02.161924 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.155277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.21.164.253:33753)
0619 02:50:02.155320 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.155322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.155733 (+ 411us) server_negotiation.cc:407] Connection header received
0619 02:50:02.155827 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.155828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.155838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.155866 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.156303 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.157065 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.158116 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.158422 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.158455 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.159451 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.159453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.159453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.159460 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.160770 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.161175 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.161176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.161176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.161236 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.161448 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.161449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.161450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.161577 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.161642 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.161794 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.161859 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:50:02.162001 29003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.155344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:33753 (local address 127.0.0.1:60286)
0619 02:50:02.155675 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.155684 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.155753 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.155924 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.155925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.156160 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.156221 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.156222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.157141 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.157142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.158029 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.158031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.158517 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.158517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.158577 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.159014 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.159019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.159363 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.160843 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.160844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.160845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.161090 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.161300 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.161301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.161301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.161367 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.161646 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.161647 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.161702 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.161776 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.161883 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":32,"thread_start_us":90,"threads_started":1}
I20260619 02:50:02.183980 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 28322
I20260619 02:50:02.184777 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.155245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.21.164.254:41601)
0619 02:50:02.155292 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.155294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.155569 (+ 275us) server_negotiation.cc:407] Connection header received
0619 02:50:02.155667 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.155668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.155679 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.155708 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.156199 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.157029 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.158189 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.158482 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.158522 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.181974 (+ 23452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.181976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.181977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.181985 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.183614 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.183954 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.183956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.183956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.184009 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.184226 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.184227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.184227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.184343 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.184377 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.184567 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.184683 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:50:02.185165 29002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.155208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52152)
0619 02:50:02.155499 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.155511 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.155595 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.155771 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.155772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.156039 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.156109 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.156112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.157224 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.157225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.158091 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.158093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.158597 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.158597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.158661 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.181309 (+ 22648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.181314 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.181817 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.183697 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.183698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.183699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.183875 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.184125 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.184126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.184126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.184171 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.184412 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.184413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.184463 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.184948 (+ 485us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.185061 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":220,"threads_started":1}
I20260619 02:50:02.186493 29004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.155571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:37545 (local address 127.0.0.1:50994)
0619 02:50:02.155676 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.155684 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.155751 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.155913 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.155914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.156155 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.156222 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.156223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.157106 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.157106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.158029 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.158031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.158500 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.158501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.158568 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.181265 (+ 22697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.181272 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.181765 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.183511 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.183512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.183512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.183696 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.183874 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.183875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.183875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.183921 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.186084 (+ 2163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.186085 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.186139 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.186211 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.186317 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":32,"thread_start_us":54,"threads_started":1}
I20260619 02:50:02.186614 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.155610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.21.164.252:37545)
0619 02:50:02.155657 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.155658 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.155730 (+ 72us) server_negotiation.cc:407] Connection header received
0619 02:50:02.155813 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.155813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.155821 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.155850 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.156296 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.157028 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.158111 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.158382 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.158419 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.181862 (+ 23443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.181864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.181864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.181871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.183408 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.183760 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.183761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.183761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.183801 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.183972 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.183973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.183973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.184097 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.184135 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.186263 (+ 2128us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.186385 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":19}
I20260619 02:50:02.189070 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:38869
--local_ip_for_outbound_sockets=127.21.164.193
--tserver_master_addrs=127.21.164.254:41601
--webserver_port=35737
--webserver_interface=127.21.164.193
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab
--principal=oryx/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.164.254:41601,127.21.164.253:33753
--tserver_master_addrs=127.21.164.254:41601,127.21.164.253:33753,127.21.164.252:37545 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837401, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.254@KRBTEST.COM for oryx/127.21.164.252@KRBTEST.COM
I20260619 02:50:02.195827 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.189763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:37545 (local address 127.0.0.1:51010)
0619 02:50:02.189844 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.189854 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.189887 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.190157 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.190158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.190348 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.190402 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.190405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.191012 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.191013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.191544 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.191546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.191856 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.191856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.191918 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.192278 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.192283 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.193473 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.194921 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.194922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.194923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.195087 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.195259 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.195259 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.195260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.195317 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.195525 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.195526 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.195572 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.195630 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.195731 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 02:50:02.195827 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.189953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.21.164.252:37545)
0619 02:50:02.190020 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.190022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.190028 (+ 6us) server_negotiation.cc:407] Connection header received
0619 02:50:02.190060 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.190060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.190069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.190110 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.190483 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.190940 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.191620 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.191801 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.191852 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.193550 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.193552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.193552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.193560 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.194824 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.195146 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.195147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.195148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.195197 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.195364 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.195364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.195365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.195472 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.195525 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.195657 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.195730 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:50:02.196300 28260 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.164.252:37545: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6"
member_type: NON_VOTER
last_known_addr {
host: "127.21.164.252"
port: 37545
}
attrs {
promote: true
}
}
dest_uuid: "1c8a057eee6c4376b5c1a06a6154abd0"
cas_config_opid_index: 5
I20260619 02:50:02.196547 28260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } }
I20260619 02:50:02.197062 28820 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 71dc4518362c47f2a5b5374fd3693ce6 (127.21.164.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.197966 28820 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.164.252:37545
I20260619 02:50:02.198038 28820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.198105 28820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:02.198522 28954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:02.199519 28954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 FOLLOWER]: Refusing update from remote peer 1c8a057eee6c4376b5c1a06a6154abd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 02:50:02.199818 28912 master.cc:537] Master@127.21.164.252:37545 shutting down...
I20260619 02:50:02.201134 28819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:02.201870 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.197163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:33753 (local address 127.0.0.1:60292)
0619 02:50:02.197250 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.197260 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.197295 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.198550 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.198552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.198554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:02.198595 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.198596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.199457 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.199457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.200696 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.200699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.201606 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.201607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.201672 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.201696 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.201747 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.201786 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 02:50:02.202066 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.197382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.21.164.253:33753)
0619 02:50:02.198388 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.198390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.198398 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:02.198421 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.198422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.198444 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.198497 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:02.198682 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.199369 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.200819 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.201418 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.201488 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.201571 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.201965 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.202007 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980}
W20260619 02:50:02.202098 28246 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 -> Peer 71dc4518362c47f2a5b5374fd3693ce6 (127.21.164.252:37545): Couldn't send request to peer 71dc4518362c47f2a5b5374fd3693ce6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:50:02.202260 28868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 LEARNER]: Refusing update from remote peer 1c8a057eee6c4376b5c1a06a6154abd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 02:50:02.202518 28819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 02:50:02.203071 28868 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 71dc4518362c47f2a5b5374fd3693ce6 (127.21.164.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.203881 28912 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:02.203972 28912 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:02.204022 28912 tablet_replica.cc:330] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: stopping tablet replica
I20260619 02:50:02.204167 29008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c8a057eee6c4376b5c1a06a6154abd0. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.204185 28897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: NON_VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: true } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.204346 28897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:50:02.204317 29008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:50:02.204576 28819 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0: attempting to promote NON_VOTER 868b192a035540c49640f0e4caa8e472 to VOTER
I20260619 02:50:02.204813 28819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } }
W20260619 02:50:02.205714 28929 connection.cc:441] server connection from 127.0.0.1:51010 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51010 (request call id 4) could send its response
I20260619 02:50:02.206162 28868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 LEARNER]: Refusing update from remote peer 1c8a057eee6c4376b5c1a06a6154abd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 02:50:02.206202 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.206064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:37545 (local address 127.0.0.1:51020)
0619 02:50:02.206122 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.206166 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:37545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 02:50:02.206352 29006 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260619 02:50:02.206569 28246 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 -> Peer 71dc4518362c47f2a5b5374fd3693ce6 (127.21.164.252:37545): Couldn't send request to peer 71dc4518362c47f2a5b5374fd3693ce6. Status: Network error: Client connection negotiation failed: client connection to 127.21.164.252:37545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 02:50:02.207233 28820 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 868b192a035540c49640f0e4caa8e472 (127.21.164.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.207389 28868 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 868b192a035540c49640f0e4caa8e472 (127.21.164.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.208168 28820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.208238 28820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:02.208307 29006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.208392 29006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:02.209033 29008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.209125 29008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:02.220175 28912 master.cc:552] Master@127.21.164.252:37545 shutdown complete.
I20260619 02:50:02.220242 28912 master_runner.cc:305] Clearing existing system tablet
I20260619 02:50:02.220379 28912 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:02.220434 28912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:02.221271 28912 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:50:02.221781 29010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.221892 28912 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:02.221935 28912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.221953 28912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/wal
uuid: "71dc4518362c47f2a5b5374fd3693ce6"
format_stamp: "Formatted at 2026-06-19 02:50:02 on dist-test-slave-jt0t"
server_key: "779cf8345aeca3596fa061598c835e8d"
server_key_iv: "2b9fbc39433005ee4874eebca295473e"
server_key_version: "encryptionkey@0"
I20260619 02:50:02.222664 28912 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:02.224187 28912 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:50:02.224255 28912 log.cc:1196] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 02:50:02.224440 28912 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: Deleting consensus metadata
I20260619 02:50:02.224548 28912 master_runner.cc:315] Copying system tablet from 127.21.164.254:41601
I20260619 02:50:02.225176 28912 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.164.254:41601
I20260619 02:50:02.232584 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.225539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.21.164.254:41601)
0619 02:50:02.225586 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.225588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.225684 (+ 96us) server_negotiation.cc:407] Connection header received
0619 02:50:02.225738 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.225739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.225749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.225819 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.226178 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.226949 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.228008 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.228308 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.228373 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.229510 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.229512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.229512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.229520 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.231309 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.231702 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.231704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.231704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.231785 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.231986 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.231987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.231988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.232122 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.232183 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.232402 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.232487 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:50:02.232615 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.225472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52156)
0619 02:50:02.225607 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.225618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.225684 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.225823 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.225824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.226040 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.226102 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.226105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.227049 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.227051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.227915 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.227919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.228407 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.228409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.228475 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.229001 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.229006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.229410 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.231389 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.231390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.231391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.231621 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.231851 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.231852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.231852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.231924 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.232186 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.232187 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.232247 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.232370 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.232490 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260619 02:50:02.233381 28280 tablet_copy_service.cc:140] P 1c8a057eee6c4376b5c1a06a6154abd0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 71dc4518362c47f2a5b5374fd3693ce6 ({username='oryx', principal='oryx/127.21.164.252@KRBTEST.COM'} at 127.0.0.1:52156)
I20260619 02:50:02.233449 28280 tablet_copy_service.cc:161] P 1c8a057eee6c4376b5c1a06a6154abd0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 71dc4518362c47f2a5b5374fd3693ce6 at {username='oryx', principal='oryx/127.21.164.252@KRBTEST.COM'} at 127.0.0.1:52156: session id = 71dc4518362c47f2a5b5374fd3693ce6-00000000000000000000000000000000
I20260619 02:50:02.234082 28280 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:50:02.234602 28912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:50:02.236184 28912 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: tablet copy: Starting download of 0 data blocks...
I20260619 02:50:02.236326 28912 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: tablet copy: Starting download of 1 WAL segments...
I20260619 02:50:02.237236 28912 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:50:02.238266 28280 tablet_copy_service.cc:342] P 1c8a057eee6c4376b5c1a06a6154abd0: Request end of tablet copy session 71dc4518362c47f2a5b5374fd3693ce6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.164.252@KRBTEST.COM'} at 127.0.0.1:52156
I20260619 02:50:02.238325 28280 tablet_copy_service.cc:433] P 1c8a057eee6c4376b5c1a06a6154abd0: ending tablet copy session 71dc4518362c47f2a5b5374fd3693ce6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 71dc4518362c47f2a5b5374fd3693ce6
I20260619 02:50:02.240720 28912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:02.242040 29018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:02.242084 29017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:02.245800 28912 server_base.cc:1034] running on GCE node
W20260619 02:50:02.245762 29020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:02.246019 28912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:02.246191 28912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:02.247321 28912 hybrid_clock.cc:648] HybridClock initialized: now 1781837402247313 us; error 30 us; skew 500 ppm
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:02.248979 28912 init.cc:377] Logged in from keytab as oryx/127.21.164.252@KRBTEST.COM (short username oryx)
I20260619 02:50:02.249289 28912 webserver.cc:460] Webserver started at http://127.21.164.252:42667/ using document root <none> and password file <none>
I20260619 02:50:02.249398 28912 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:02.249430 28912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:02.250216 28912 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:02.250777 29027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.250947 28912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.250995 28912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/wal
uuid: "71dc4518362c47f2a5b5374fd3693ce6"
format_stamp: "Formatted at 2026-06-19 02:50:02 on dist-test-slave-jt0t"
server_key: "779cf8345aeca3596fa061598c835e8d"
server_key_iv: "2b9fbc39433005ee4874eebca295473e"
server_key_version: "encryptionkey@0"
I20260619 02:50:02.251053 28912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:02.284618 28912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:02.286434 28912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:02.289582 28912 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.252:37545
I20260619 02:50:02.290064 28912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
I20260619 02:50:02.290970 29080 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:50:02.291479 29080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: Bootstrap starting.
I20260619 02:50:02.293088 29079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.252:37545 every 8 connection(s)
I20260619 02:50:02.294502 29080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:50:02.294708 29080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: Bootstrap complete.
I20260619 02:50:02.294901 29080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } }
I20260619 02:50:02.294988 29080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 71dc4518362c47f2a5b5374fd3693ce6, State: Initialized, Role: LEARNER
I20260619 02:50:02.295024 29080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } }
I20260619 02:50:02.295228 29080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:02.295218 29083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.295416 29083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:50:02.296640 29094 catalog_manager.cc:1259] Loaded cluster ID: c4c10474c1dd4db7a5cc6420f8dc823c
I20260619 02:50:02.296757 29094 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: loading cluster ID for follower catalog manager: success
I20260619 02:50:02.298118 29094 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: acquiring CA information for follower catalog manager: success
I20260619 02:50:02.298442 29094 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 02:50:02.303017 29005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:02.303238 29005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:02.303336 29005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:02.304593 29005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:02.304638 29005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:02.304702 29005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:50:02.306360 29005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab
--principal=oryx/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:38869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=35737
--tserver_master_addrs=127.21.164.254:41601,127.21.164.253:33753,127.21.164.252:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:02.306605 29005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:02.306833 29005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:02.312237 29099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:02.312312 29005 server_base.cc:1034] running on GCE node
W20260619 02:50:02.312251 29102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:02.312278 29100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:02.312638 29005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:02.312927 29005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:02.314093 29005 hybrid_clock.cc:648] HybridClock initialized: now 1781837402314079 us; error 30 us; skew 500 ppm
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:02.316605 29005 init.cc:377] Logged in from keytab as oryx/127.21.164.193@KRBTEST.COM (short username oryx)
I20260619 02:50:02.317093 29005 webserver.cc:460] Webserver started at http://127.21.164.193:35737/ using document root <none> and password file <none>
I20260619 02:50:02.317320 29005 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:02.317386 29005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:02.318794 29005 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:50:02.319458 29109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.319674 29005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:50:02.319777 29005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "33bc48e3da76477e9a8136915ad87b63"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "b55e535844a9bfaa7533f52838fb7854"
server_key_iv: "6b2215dac3e622fad354695c2a77e8f3"
server_key_version: "encryptionkey@0"
I20260619 02:50:02.320142 29005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:02.343719 29005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:02.346302 29005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:02.346467 29005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:02.346705 29005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:02.347033 29005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:02.347066 29005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.347096 29005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:02.347113 29005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.354151 29005 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:38869
I20260619 02:50:02.354218 29222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:38869 every 8 connection(s)
I20260619 02:50:02.354537 29005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.252@KRBTEST.COM
I20260619 02:50:02.362112 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 29005
I20260619 02:50:02.362262 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 28456
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.253@KRBTEST.COM
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:02.366647 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.355609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:35601 (local address 127.21.164.252:37545)
0619 02:50:02.355704 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.355706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.355716 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:02.356619 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.356620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.356633 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.356690 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.357590 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.358563 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.359819 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.360123 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.360176 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.362846 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.362848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.362848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.362857 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.364425 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.365541 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.365542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.365543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.365647 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.365903 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.365904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.365904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.366039 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.366082 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.366461 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.366546 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":44,"threads_started":1}
I20260619 02:50:02.366645 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.355233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:47499 (local address 127.21.164.253:33753)
0619 02:50:02.355292 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.355294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.355529 (+ 235us) server_negotiation.cc:407] Connection header received
0619 02:50:02.356619 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.356621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.356633 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.356688 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.357566 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.358358 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.359796 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.360001 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.360034 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.363571 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.363573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.363573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.363581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.364984 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.365542 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.365542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.365543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.365616 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.365903 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.365904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.365904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.366046 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.366082 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.366461 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.366544 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:50:02.366899 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.355886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:54301 (local address 127.21.164.254:41601)
0619 02:50:02.355945 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.355947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.355955 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:02.356601 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.356602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.356614 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.356693 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.358472 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.358987 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.359762 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.360045 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.360108 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.364105 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.364106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.364107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.364116 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.365809 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.366173 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.366174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.366175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.366227 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.366390 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.366390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.366391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.366510 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.366545 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.366716 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.366795 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:50:02.367200 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.355529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:37545 (local address 127.21.164.193:35601)
0619 02:50:02.355641 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.355650 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.356504 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.356789 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.356794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.357205 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.357477 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.357483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.358642 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.358643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.359686 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.359688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.360187 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.360188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.360263 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.360922 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.360932 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.362732 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.365057 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.365064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.365066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.365452 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.365728 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.365728 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.365729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.365826 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.366126 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.366133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.366305 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.366752 (+ 447us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.366952 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":77,"thread_start_us":64,"threads_started":1}
I20260619 02:50:02.367203 29227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.355238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:33753 (local address 127.21.164.193:47499)
0619 02:50:02.355410 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.355445 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.356499 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.356789 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.356794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.357194 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.357477 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.357483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.358475 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.358476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.359686 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.359688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.360110 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.360111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.360263 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.361025 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.361029 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.363465 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.365055 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.365064 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.365066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.365452 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.365687 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.365688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.365688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.365826 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.366126 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.366133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.366309 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.366735 (+ 426us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.367012 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":130,"thread_start_us":100,"threads_started":1}
I20260619 02:50:02.367211 29230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.355700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.193:54301)
0619 02:50:02.355860 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.355869 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.356507 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.356788 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.356807 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.357194 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.357476 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.357481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.359059 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.359060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.359682 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.359684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.360118 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.360118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.360263 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.360949 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.360953 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.363602 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.365909 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.365909 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.365910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.366099 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.366286 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.366286 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.366286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.366333 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.366582 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.366583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.366621 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.367003 (+ 382us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.367103 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":9,"thread_start_us":64,"threads_started":1}
I20260619 02:50:02.367642 29225 heartbeater.cc:347] Connected to a master server at 127.21.164.253:33753
I20260619 02:50:02.367667 29223 heartbeater.cc:347] Connected to a master server at 127.21.164.254:41601
I20260619 02:50:02.367727 29223 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:02.367728 29225 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:02.367885 29223 heartbeater.cc:511] Master 127.21.164.254:41601 requested a full tablet report, sending...
I20260619 02:50:02.367899 29225 heartbeater.cc:511] Master 127.21.164.253:33753 requested a full tablet report, sending...
I20260619 02:50:02.368305 28260 ts_manager.cc:194] Re-registered known tserver with Master: 33bc48e3da76477e9a8136915ad87b63 (127.21.164.193:38869)
I20260619 02:50:02.368372 28858 ts_manager.cc:194] Registered new tserver with Master: 33bc48e3da76477e9a8136915ad87b63 (127.21.164.193:38869)
I20260619 02:50:02.368713 29224 heartbeater.cc:347] Connected to a master server at 127.21.164.252:37545
I20260619 02:50:02.368754 29224 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:02.368806 29224 heartbeater.cc:511] Master 127.21.164.252:37545 requested a full tablet report, sending...
I20260619 02:50:02.368826 28260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:54301
I20260619 02:50:02.369166 29044 ts_manager.cc:194] Registered new tserver with Master: 33bc48e3da76477e9a8136915ad87b63 (127.21.164.193:38869)
I20260619 02:50:02.370604 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:40493
--local_ip_for_outbound_sockets=127.21.164.194
--tserver_master_addrs=127.21.164.254:41601
--webserver_port=39051
--webserver_interface=127.21.164.194
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab
--principal=oryx/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.164.254:41601,127.21.164.253:33753
--tserver_master_addrs=127.21.164.254:41601,127.21.164.253:33753,127.21.164.252:37545 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:02.452111 29231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:02.452337 29231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:02.452361 29231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:02.453601 29231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:02.453641 29231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:02.453667 29231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:50:02.455174 29231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab
--principal=oryx/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:40493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=39051
--tserver_master_addrs=127.21.164.254:41601,127.21.164.253:33753,127.21.164.252:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:02.455379 29231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:02.455574 29231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:02.460434 29237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:02.460490 29236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:02.460466 29239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:02.460610 29231 server_base.cc:1034] running on GCE node
I20260619 02:50:02.460896 29231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:02.461115 29231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:02.462273 29231 hybrid_clock.cc:648] HybridClock initialized: now 1781837402462253 us; error 36 us; skew 500 ppm
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:02.464854 29231 init.cc:377] Logged in from keytab as oryx/127.21.164.194@KRBTEST.COM (short username oryx)
I20260619 02:50:02.465303 29231 webserver.cc:460] Webserver started at http://127.21.164.194:39051/ using document root <none> and password file <none>
I20260619 02:50:02.465507 29231 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:02.465569 29231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:02.466964 29231 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:02.467588 29246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.467742 29231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:02.467826 29231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "ca7f70e8e0f340a5b488afc254535a6a"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "87ee628cc8ff81e87155645dc584498d"
server_key_iv: "6d02311eeb40fc21e5106a83f0152077"
server_key_version: "encryptionkey@0"
I20260619 02:50:02.468180 29231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:02.494891 29231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:02.497246 29231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:02.497393 29231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:02.497614 29231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:02.497939 29231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:02.497974 29231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.498003 29231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:02.498025 29231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.503580 29231 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:40493
I20260619 02:50:02.503630 29359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:40493 every 8 connection(s)
I20260619 02:50:02.503965 29231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:50:02.505772 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 29231
I20260619 02:50:02.505932 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 28590
I20260619 02:50:02.511847 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:46603
--local_ip_for_outbound_sockets=127.21.164.195
--tserver_master_addrs=127.21.164.254:41601
--webserver_port=46139
--webserver_interface=127.21.164.195
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab
--principal=oryx/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.164.254:41601,127.21.164.253:33753
--tserver_master_addrs=127.21.164.254:41601,127.21.164.253:33753,127.21.164.252:37545 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.252@KRBTEST.COM
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.253@KRBTEST.COM
I20260619 02:50:02.517912 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.504689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:45797 (local address 127.21.164.252:37545)
0619 02:50:02.504735 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.504738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.504983 (+ 245us) server_negotiation.cc:407] Connection header received
0619 02:50:02.506452 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.506454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.506467 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.506507 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.507363 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.508147 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.509799 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.510113 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.510154 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.513702 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.513704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.513704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.513712 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.515433 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.516171 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.516173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.516173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.516299 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.516740 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.516765 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.516765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.516952 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.517034 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.517645 (+ 611us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.517768 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:50:02.518049 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.505154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:52137 (local address 127.21.164.254:41601)
0619 02:50:02.505208 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.505209 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.505219 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:02.505723 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.505725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.505737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.505787 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.507362 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.508133 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.509789 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.510067 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.510110 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.513047 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.513048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.513049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.513058 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.514999 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.517228 (+ 2229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.517231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.517231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.517295 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.517510 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.517511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.517512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.517639 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.517687 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.517870 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.517955 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:50:02.518606 29366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.504990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.194:52137)
0619 02:50:02.505146 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.505155 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.505476 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.505814 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.505814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.506939 (+ 1125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.507250 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.507256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.508426 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.508427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.509693 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.509695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.510200 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.510201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.510335 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.510955 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.510966 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.512920 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.515184 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.515193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.515195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.515515 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.517367 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.517368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.517368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.517438 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.517719 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.517719 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.517758 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.518165 (+ 407us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.518342 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":68,"thread_start_us":58,"threads_started":1}
I20260619 02:50:02.518781 29365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.504684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:37545 (local address 127.21.164.194:45797)
0619 02:50:02.504863 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.504901 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.505482 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.506595 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.506596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.506934 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.507250 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.507257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.508282 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.508283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.509695 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.509698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.510232 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.510233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.510336 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.511063 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.511068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.513594 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.515506 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.515507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.515535 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.515833 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.516403 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.516404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.516404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.516607 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.517069 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.517077 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.517408 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.518092 (+ 684us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.518666 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":392,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260619 02:50:02.519160 29360 heartbeater.cc:347] Connected to a master server at 127.21.164.254:41601
I20260619 02:50:02.519326 29360 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:02.519418 29361 heartbeater.cc:347] Connected to a master server at 127.21.164.252:37545
I20260619 02:50:02.519472 29361 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:02.519593 29361 heartbeater.cc:511] Master 127.21.164.252:37545 requested a full tablet report, sending...
I20260619 02:50:02.519598 29360 heartbeater.cc:511] Master 127.21.164.254:41601 requested a full tablet report, sending...
I20260619 02:50:02.519981 28260 ts_manager.cc:194] Re-registered known tserver with Master: ca7f70e8e0f340a5b488afc254535a6a (127.21.164.194:40493)
I20260619 02:50:02.520002 29044 ts_manager.cc:194] Registered new tserver with Master: ca7f70e8e0f340a5b488afc254535a6a (127.21.164.194:40493)
I20260619 02:50:02.520494 28260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:52137
I20260619 02:50:02.522199 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.504717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:42553 (local address 127.21.164.253:33753)
0619 02:50:02.504763 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.504766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.504977 (+ 211us) server_negotiation.cc:407] Connection header received
0619 02:50:02.505581 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.505583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.505597 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.505659 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.513536 (+ 7877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.514264 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.515283 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.515568 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.515608 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.518130 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.518132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.518132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.518141 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.520504 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.521058 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.521060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.521060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.521153 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.521433 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.521434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.521435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.521643 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.521698 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.522003 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.522106 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:50:02.522701 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.504684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:33753 (local address 127.21.164.194:42553)
0619 02:50:02.504852 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.504901 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.505463 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.505710 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.505714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.513363 (+ 7649us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.513428 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.513430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.514362 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.514363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.515196 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.515198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.515683 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.515683 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.515744 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.516133 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.516138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.518032 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.520613 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.520613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.520614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.520944 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.521246 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.521247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.521247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.521333 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.521753 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.521753 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.521824 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.522282 (+ 458us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.522397 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260619 02:50:02.523288 29362 heartbeater.cc:347] Connected to a master server at 127.21.164.253:33753
I20260619 02:50:02.523350 29362 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:02.523427 29362 heartbeater.cc:511] Master 127.21.164.253:33753 requested a full tablet report, sending...
I20260619 02:50:02.523743 28858 ts_manager.cc:194] Registered new tserver with Master: ca7f70e8e0f340a5b488afc254535a6a (127.21.164.194:40493)
W20260619 02:50:02.600804 29367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:02.601012 29367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:02.601034 29367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:02.602178 29367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:02.602217 29367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:02.602243 29367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:50:02.603706 29367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab
--principal=oryx/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:46603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=46139
--tserver_master_addrs=127.21.164.254:41601,127.21.164.253:33753,127.21.164.252:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:02.603915 29367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:02.604103 29367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:02.609031 29372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:02.609088 29367 server_base.cc:1034] running on GCE node
W20260619 02:50:02.609090 29373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:02.609036 29375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:02.609532 29367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:02.609784 29367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:02.610960 29367 hybrid_clock.cc:648] HybridClock initialized: now 1781837402610945 us; error 32 us; skew 500 ppm
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:02.613412 29367 init.cc:377] Logged in from keytab as oryx/127.21.164.195@KRBTEST.COM (short username oryx)
I20260619 02:50:02.613857 29367 webserver.cc:460] Webserver started at http://127.21.164.195:46139/ using document root <none> and password file <none>
I20260619 02:50:02.614077 29367 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:02.614140 29367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:02.615551 29367 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:02.616176 29382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.616417 29367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:02.616509 29367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "1c38c65a824845b4b92ab1da928b5f06"
format_stamp: "Formatted at 2026-06-19 02:50:01 on dist-test-slave-jt0t"
server_key: "86dffdb40ccb8cf32e16aa051e2d5680"
server_key_iv: "67c19854e3b0250fffa869c488074e7d"
server_key_version: "encryptionkey@0"
I20260619 02:50:02.616914 29367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:02.642956 29367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:02.645164 29367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:02.645308 29367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:02.645541 29367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:02.645841 29367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:02.645876 29367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.645906 29367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:02.645924 29367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:02.651347 29367 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:46603
I20260619 02:50:02.651453 29495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:46603 every 8 connection(s)
I20260619 02:50:02.651739 29367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.252@KRBTEST.COM
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.253@KRBTEST.COM
Jun 19 02:50:02 dist-test-slave-jt0t krb5kdc[28215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837402, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:02.659309 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 29367
I20260619 02:50:02.661880 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.652686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:37545 (local address 127.21.164.195:49847)
0619 02:50:02.652787 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.652796 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.653250 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.653492 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.653497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.654020 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.654204 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.654210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.655133 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.655134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.656000 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.656001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.656353 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.656354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.656443 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.656957 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.656967 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.658545 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.660358 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.660366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.660369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.660697 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.660894 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.660895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.660895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.661029 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.661276 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.661283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.661439 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.661548 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.661697 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":32,"thread_start_us":56,"threads_started":1}
I20260619 02:50:02.662159 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.653048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:49107 (local address 127.21.164.253:33753)
0619 02:50:02.653095 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.653097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.653104 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:50:02.653337 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.653338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.653349 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.653405 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.654298 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.655060 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.655937 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.656167 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.656224 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.659473 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.659475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.659476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.659484 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.660998 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.661353 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.661354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.661354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.661429 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.661620 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.661621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.661621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.661751 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.661792 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.661987 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.662064 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:50:02.662431 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.652735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:49847 (local address 127.21.164.252:37545)
0619 02:50:02.652778 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.652780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.652841 (+ 61us) server_negotiation.cc:407] Connection header received
0619 02:50:02.653369 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.653370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.653382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.653413 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.654324 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.655053 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.656090 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.656294 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.656347 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.658633 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.658634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.658634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.658641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.660271 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.660759 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.660760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.660760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.660822 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.661103 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.661104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.661105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.661223 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.661278 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.662260 (+ 982us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.662341 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:50:02.662482 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.652864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:33753 (local address 127.21.164.195:49107)
0619 02:50:02.652983 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.652992 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.653252 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.653748 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.653749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.653990 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.654202 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.654207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.655162 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.655162 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.655852 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.655854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.656286 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.656287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.656440 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.657147 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.657151 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.659340 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.661078 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.661079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.661080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.661272 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.661498 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.661499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.661500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.661551 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.661835 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.661836 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.661876 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.662276 (+ 400us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.662369 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":36,"thread_start_us":67,"threads_started":1}
I20260619 02:50:02.662784 29497 heartbeater.cc:347] Connected to a master server at 127.21.164.252:37545
I20260619 02:50:02.662806 29498 heartbeater.cc:347] Connected to a master server at 127.21.164.253:33753
I20260619 02:50:02.662946 29497 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:02.662945 29498 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:02.663175 29497 heartbeater.cc:511] Master 127.21.164.252:37545 requested a full tablet report, sending...
I20260619 02:50:02.663175 29498 heartbeater.cc:511] Master 127.21.164.253:33753 requested a full tablet report, sending...
I20260619 02:50:02.663618 28858 ts_manager.cc:194] Registered new tserver with Master: 1c38c65a824845b4b92ab1da928b5f06 (127.21.164.195:46603)
I20260619 02:50:02.663617 29044 ts_manager.cc:194] Registered new tserver with Master: 1c38c65a824845b4b92ab1da928b5f06 (127.21.164.195:46603)
I20260619 02:50:02.665047 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.652416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:39143 (local address 127.21.164.254:41601)
0619 02:50:02.652501 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.652503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.652704 (+ 201us) server_negotiation.cc:407] Connection header received
0619 02:50:02.653369 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.653370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.653390 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.653420 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.654311 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.654803 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.656063 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.656328 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.656386 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.660448 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.660450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.660450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.660458 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.662153 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.664234 (+ 2081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.664235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.664236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.664298 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.664490 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.664491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.664491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.664616 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.664683 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.664872 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.664957 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:50:02.665355 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.652402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.195:39143)
0619 02:50:02.652552 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.652589 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.653245 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.653492 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.653497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.654001 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:02.654204 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.654210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.654899 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.654900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.655984 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.655986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.656434 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.656435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.656491 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.657235 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:02.657239 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:02.660346 (+ 3107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:02.662790 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.662791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.662791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.664157 (+ 1366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.664368 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:02.664369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:02.664369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:02.664426 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:02.664729 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:02.664729 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:02.664768 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.665153 (+ 385us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.665251 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":25,"thread_start_us":74,"threads_started":1}
I20260619 02:50:02.665621 29496 heartbeater.cc:347] Connected to a master server at 127.21.164.254:41601
I20260619 02:50:02.665671 29496 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:02.665737 29496 heartbeater.cc:511] Master 127.21.164.254:41601 requested a full tablet report, sending...
I20260619 02:50:02.665956 28260 ts_manager.cc:194] Re-registered known tserver with Master: 1c38c65a824845b4b92ab1da928b5f06 (127.21.164.195:46603)
I20260619 02:50:02.666409 28260 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:39143
I20260619 02:50:02.671104 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.662993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.21.164.254:41601)
0619 02:50:02.663777 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.663778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.663922 (+ 144us) server_negotiation.cc:407] Connection header received
0619 02:50:02.663945 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.663945 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.663957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.664014 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.664741 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.665437 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.666535 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.666806 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.666861 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.668588 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.668589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.668589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.668595 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.669999 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.670294 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.670295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.670295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.670347 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.670517 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.670517 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.670518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.670633 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.670724 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.670831 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.670912 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":79,"threads_started":1}
I20260619 02:50:02.671474 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.663356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.21.164.252:37545)
0619 02:50:02.663399 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.663401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.663959 (+ 558us) server_negotiation.cc:407] Connection header received
0619 02:50:02.664043 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.664044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.664053 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.664079 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.664697 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.665409 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.666543 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.666808 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.666845 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.668371 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.668373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.668373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.668379 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.670167 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.670651 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.670652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.670653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.670705 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.670911 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.670912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.670912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.671045 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.671109 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.671319 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.671389 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:50:02.671588 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.660569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.21.164.253:33753)
0619 02:50:02.660703 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.660705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.663718 (+ 3013us) server_negotiation.cc:407] Connection header received
0619 02:50:02.663740 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.663741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.663753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.663808 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.664635 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.665874 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.667044 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.667307 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.667378 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.668525 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.668526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.668526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.668533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.670255 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.670640 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.670641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.670642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.670694 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.670875 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.670875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.670876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.670993 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.671385 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.671422 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.671496 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260619 02:50:02.672878 28260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:50:02.674340 28260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:50:02.681248 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.677725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:40493 (local address 127.0.0.1:35150)
0619 02:50:02.677889 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.677898 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.677933 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.678313 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.678314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.678316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:02.678374 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.678376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.679289 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.679290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.680044 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.680045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.681015 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.681016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.681074 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.681079 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.681126 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.681165 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260619 02:50:02.681258 29519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.677982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.21.164.194:40493)
0619 02:50:02.678167 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.678173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.678183 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:02.678220 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.678223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.678239 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.678317 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:02.678430 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.679217 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.680180 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.680943 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.681024 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.681078 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.681137 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.681172 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260619 02:50:02.681882 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.677709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:38869 (local address 127.0.0.1:50130)
0619 02:50:02.677751 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.677761 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.677805 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.678527 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.678527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.678529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:02.678555 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.678556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.679989 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.679990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.680853 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.680855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.681657 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.681658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.681716 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.681719 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.681771 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.681809 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":40}
I20260619 02:50:02.681892 29518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.677813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.21.164.193:38869)
0619 02:50:02.678172 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.678177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.678187 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:02.678226 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.678228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.678243 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.678526 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:02.678606 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.679934 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.680940 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.681588 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.681666 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.681722 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.681784 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.681817 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":89,"threads_started":1}
I20260619 02:50:02.682344 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.677800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:46603 (local address 127.0.0.1:37744)
0619 02:50:02.678350 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.678359 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.678437 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.678644 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.678645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.678647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:02.678686 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.678688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.679937 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.679938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.681169 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.681171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.682121 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.682122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.682180 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.682184 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.682232 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.682267 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":83,"thread_start_us":100,"threads_started":1}
I20260619 02:50:02.682312 29294 tablet_service.cc:1467] Processing CreateTablet for tablet ffbb67e09c8a40b9a6ca13f46991e638 (DEFAULT_TABLE table=test-table [id=a17d1abfbaea41bb86978942b46c9635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:02.682646 29294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffbb67e09c8a40b9a6ca13f46991e638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:02.682559 29157 tablet_service.cc:1467] Processing CreateTablet for tablet ffbb67e09c8a40b9a6ca13f46991e638 (DEFAULT_TABLE table=test-table [id=a17d1abfbaea41bb86978942b46c9635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:02.682906 29157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffbb67e09c8a40b9a6ca13f46991e638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:02.682911 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.678041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.21.164.195:46603)
0619 02:50:02.678369 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.678372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.678429 (+ 57us) server_negotiation.cc:407] Connection header received
0619 02:50:02.678506 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.678508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.678521 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.678584 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:02.678756 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.679845 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.681254 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.681959 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.682042 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.682088 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.682273 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.682318 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
I20260619 02:50:02.683517 29430 tablet_service.cc:1467] Processing CreateTablet for tablet ffbb67e09c8a40b9a6ca13f46991e638 (DEFAULT_TABLE table=test-table [id=a17d1abfbaea41bb86978942b46c9635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:02.683804 29430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffbb67e09c8a40b9a6ca13f46991e638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:02.686226 29522 tablet_bootstrap.cc:492] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: Bootstrap starting.
I20260619 02:50:02.686753 29522 tablet_bootstrap.cc:654] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:02.686842 29521 tablet_bootstrap.cc:492] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63: Bootstrap starting.
I20260619 02:50:02.687024 29522 log.cc:824] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:02.687436 29521 tablet_bootstrap.cc:654] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:02.687687 29521 log.cc:824] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:02.687697 29522 tablet_bootstrap.cc:492] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: No bootstrap required, opened a new log
I20260619 02:50:02.687752 29522 ts_tablet_manager.cc:1397] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:50:02.688362 29521 tablet_bootstrap.cc:492] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63: No bootstrap required, opened a new log
I20260619 02:50:02.688412 29521 ts_tablet_manager.cc:1397] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:50:02.688967 29524 tablet_bootstrap.cc:492] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06: Bootstrap starting.
I20260619 02:50:02.689420 29522 raft_consensus.cc:348] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } }
I20260619 02:50:02.689556 29524 tablet_bootstrap.cc:654] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:02.689594 29522 raft_consensus.cc:374] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:02.689627 29522 raft_consensus.cc:729] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7f70e8e0f340a5b488afc254535a6a, State: Initialized, Role: FOLLOWER
I20260619 02:50:02.689757 29522 consensus_queue.cc:260] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } }
I20260619 02:50:02.689851 29524 log.cc:824] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:02.690006 29522 ts_tablet_manager.cc:1428] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:50:02.689960 29521 raft_consensus.cc:348] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } }
I20260619 02:50:02.690088 29521 raft_consensus.cc:374] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:02.690126 29360 heartbeater.cc:503] Master 127.21.164.254:41601 was elected leader, sending a full tablet report...
I20260619 02:50:02.690129 29521 raft_consensus.cc:729] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33bc48e3da76477e9a8136915ad87b63, State: Initialized, Role: FOLLOWER
I20260619 02:50:02.690260 29521 consensus_queue.cc:260] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } }
I20260619 02:50:02.690519 29521 ts_tablet_manager.cc:1428] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:50:02.690631 29223 heartbeater.cc:503] Master 127.21.164.254:41601 was elected leader, sending a full tablet report...
I20260619 02:50:02.690866 29524 tablet_bootstrap.cc:492] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06: No bootstrap required, opened a new log
I20260619 02:50:02.690985 29524 ts_tablet_manager.cc:1397] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:50:02.693297 29524 raft_consensus.cc:348] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } }
I20260619 02:50:02.693436 29524 raft_consensus.cc:374] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:02.693457 29524 raft_consensus.cc:729] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c38c65a824845b4b92ab1da928b5f06, State: Initialized, Role: FOLLOWER
I20260619 02:50:02.693572 29524 consensus_queue.cc:260] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } }
I20260619 02:50:02.693819 29524 ts_tablet_manager.cc:1428] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 02:50:02.693866 29496 heartbeater.cc:503] Master 127.21.164.254:41601 was elected leader, sending a full tablet report...
W20260619 02:50:02.754892 29363 tablet.cc:2369] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:50:02.794045 29529 raft_consensus.cc:482] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:50:02.794171 29529 raft_consensus.cc:504] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } }
I20260619 02:50:02.794669 29529 leader_election.cc:288] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33bc48e3da76477e9a8136915ad87b63 (127.21.164.193:38869), ca7f70e8e0f340a5b488afc254535a6a (127.21.164.194:40493)
I20260619 02:50:02.797806 29519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.795027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:60473 (local address 127.21.164.194:40493)
0619 02:50:02.795067 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.795068 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.795074 (+ 6us) server_negotiation.cc:407] Connection header received
0619 02:50:02.795094 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.795094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.795101 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.795142 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:02.795337 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.795977 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.797137 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.797499 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.797562 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.797572 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.797698 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.797736 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:50:02.797808 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.794831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:40493 (local address 127.21.164.195:60473)
0619 02:50:02.794941 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.794951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.794997 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.795186 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.795187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.795189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:02.795267 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.795269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.796044 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.796044 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.797059 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.797061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.797566 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.797567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.797641 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.797645 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.797688 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.797732 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":116}
I20260619 02:50:02.797876 29518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.795313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:34811 (local address 127.21.164.193:38869)
0619 02:50:02.795355 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.795356 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.795363 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:50:02.795412 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.795413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.795420 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.795472 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:02.795622 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.796165 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.797200 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.797556 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.797616 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.797625 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.797772 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.797810 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:50:02.797941 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.794822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:38869 (local address 127.21.164.195:34811)
0619 02:50:02.794882 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.794895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.794931 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.795497 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.795498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.795499 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:02.795553 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.795555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.796238 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.796239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.797120 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.797122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.797621 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.797622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.797713 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.797716 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.797760 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.797799 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 02:50:02.798239 29177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffbb67e09c8a40b9a6ca13f46991e638" candidate_uuid: "1c38c65a824845b4b92ab1da928b5f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33bc48e3da76477e9a8136915ad87b63" is_pre_election: true
I20260619 02:50:02.798239 29314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffbb67e09c8a40b9a6ca13f46991e638" candidate_uuid: "1c38c65a824845b4b92ab1da928b5f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca7f70e8e0f340a5b488afc254535a6a" is_pre_election: true
I20260619 02:50:02.798396 29177 raft_consensus.cc:2454] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c38c65a824845b4b92ab1da928b5f06 in term 0.
I20260619 02:50:02.798396 29314 raft_consensus.cc:2454] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c38c65a824845b4b92ab1da928b5f06 in term 0.
I20260619 02:50:02.798640 29385 leader_election.cc:302] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c38c65a824845b4b92ab1da928b5f06, ca7f70e8e0f340a5b488afc254535a6a; no voters:
I20260619 02:50:02.798776 29529 raft_consensus.cc:2781] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:50:02.798830 29529 raft_consensus.cc:482] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:50:02.798853 29529 raft_consensus.cc:3037] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:02.799561 29529 raft_consensus.cc:504] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } }
I20260619 02:50:02.799702 29529 leader_election.cc:288] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [CANDIDATE]: Term 1 election: Requested vote from peers 33bc48e3da76477e9a8136915ad87b63 (127.21.164.193:38869), ca7f70e8e0f340a5b488afc254535a6a (127.21.164.194:40493)
I20260619 02:50:02.799822 29177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffbb67e09c8a40b9a6ca13f46991e638" candidate_uuid: "1c38c65a824845b4b92ab1da928b5f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33bc48e3da76477e9a8136915ad87b63"
I20260619 02:50:02.799835 29314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffbb67e09c8a40b9a6ca13f46991e638" candidate_uuid: "1c38c65a824845b4b92ab1da928b5f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca7f70e8e0f340a5b488afc254535a6a"
I20260619 02:50:02.799901 29177 raft_consensus.cc:3037] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:02.799901 29314 raft_consensus.cc:3037] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:02.800781 29177 raft_consensus.cc:2454] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c38c65a824845b4b92ab1da928b5f06 in term 1.
I20260619 02:50:02.800781 29314 raft_consensus.cc:2454] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c38c65a824845b4b92ab1da928b5f06 in term 1.
I20260619 02:50:02.800921 29385 leader_election.cc:302] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c38c65a824845b4b92ab1da928b5f06, ca7f70e8e0f340a5b488afc254535a6a; no voters:
I20260619 02:50:02.801017 29529 raft_consensus.cc:2781] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:02.801156 29529 raft_consensus.cc:686] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 1 LEADER]: Becoming Leader. State: Replica: 1c38c65a824845b4b92ab1da928b5f06, State: Running, Role: LEADER
I20260619 02:50:02.801249 29529 consensus_queue.cc:237] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } }
I20260619 02:50:02.802143 28260 catalog_manager.cc:5495] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c38c65a824845b4b92ab1da928b5f06 (127.21.164.195). New cstate: current_term: 1 leader_uuid: "1c38c65a824845b4b92ab1da928b5f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } health_report { overall_health: HEALTHY } } }
I20260619 02:50:02.804888 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.802208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:37545 (local address 127.0.0.1:51044)
0619 02:50:02.802249 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:02.802261 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:02.802297 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:02.802466 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:02.802467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:02.802469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:02.802515 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.802517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.803267 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.803269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.804094 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:02.804098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.804698 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.804698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:02.804760 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.804763 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:02.804796 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 02:50:02.804836 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 02:50:02.804888 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.802314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.21.164.252:37545)
0619 02:50:02.802354 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.802356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.802362 (+ 6us) server_negotiation.cc:407] Connection header received
0619 02:50:02.802383 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.802383 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.802404 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.802456 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:02.802578 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.803194 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.804192 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.804598 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.804703 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.804782 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.804801 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.804833 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:50:02.805239 29054 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260619 02:50:02.806098 29532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c8a057eee6c4376b5c1a06a6154abd0. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.806209 29532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:50:02.806447 28820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: NON_VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: true } } }
I20260619 02:50:02.806572 28820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:02.806959 28820 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0: attempting to promote NON_VOTER 71dc4518362c47f2a5b5374fd3693ce6 to VOTER
I20260619 02:50:02.807086 28820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } }
I20260619 02:50:02.807466 28868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Refusing update from remote peer 1c8a057eee6c4376b5c1a06a6154abd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 02:50:02.807485 29054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 LEARNER]: Refusing update from remote peer 1c8a057eee6c4376b5c1a06a6154abd0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260619 02:50:02.807646 28819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260619 02:50:02.807737 28820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260619 02:50:02.808269 28820 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 71dc4518362c47f2a5b5374fd3693ce6 (127.21.164.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:02.808470 28868 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 71dc4518362c47f2a5b5374fd3693ce6 (127.21.164.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:02.808488 29054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 71dc4518362c47f2a5b5374fd3693ce6 (127.21.164.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:02.809206 28820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:02.809288 28820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:02.809355 28820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:02.809405 28820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:02.809446 29006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:02.809422 29532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:02.809504 29006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:02.809505 29532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:02.809682 29538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:02.809751 29538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:02.827685 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.825517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.21.164.195:46603)
0619 02:50:02.825570 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.825572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.825580 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:02.825630 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.825630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.825647 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.825705 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:02.825796 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.826258 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.826981 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.827185 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.827237 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.827362 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:02.827480 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:02.827538 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.827579 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.827595 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:50:02.829721 29177 raft_consensus.cc:1261] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63 [term 1 FOLLOWER]: Refusing update from remote peer 1c38c65a824845b4b92ab1da928b5f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:02.829721 29314 raft_consensus.cc:1261] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a [term 1 FOLLOWER]: Refusing update from remote peer 1c38c65a824845b4b92ab1da928b5f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:02.830125 29530 consensus_queue.cc:1035] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:02.830267 29529 consensus_queue.cc:1035] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:02.832759 29540 mvcc.cc:205] Tried to move back new op lower bound from 7298406001987510272 to 7298406001874722816. Current Snapshot: MvccSnapshot[applied={T|T < 7298406001987510272}]
I20260619 02:50:02.834165 29541 mvcc.cc:205] Tried to move back new op lower bound from 7298406001987510272 to 7298406001874722816. Current Snapshot: MvccSnapshot[applied={T|T < 7298406001987510272}]
I20260619 02:50:02.834867 29542 mvcc.cc:205] Tried to move back new op lower bound from 7298406001987510272 to 7298406001874722816. Current Snapshot: MvccSnapshot[applied={T|T < 7298406001987510272 or (T in {7298406001987510272})}]
I20260619 02:50:02.836298 28259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:02.836416 28259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:02.838012 28259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c8a057eee6c4376b5c1a06a6154abd0: Sending DeleteTablet for 3 replicas of tablet ffbb67e09c8a40b9a6ca13f46991e638
I20260619 02:50:02.838330 29157 tablet_service.cc:1514] Processing DeleteTablet for tablet ffbb67e09c8a40b9a6ca13f46991e638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:02 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:50130
I20260619 02:50:02.838364 29294 tablet_service.cc:1514] Processing DeleteTablet for tablet ffbb67e09c8a40b9a6ca13f46991e638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:02 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:35150
I20260619 02:50:02.838362 29430 tablet_service.cc:1514] Processing DeleteTablet for tablet ffbb67e09c8a40b9a6ca13f46991e638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:02 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:37744
I20260619 02:50:02.838582 29549 tablet_replica.cc:330] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63: stopping tablet replica
I20260619 02:50:02.838610 29550 tablet_replica.cc:330] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: stopping tablet replica
I20260619 02:50:02.838690 29550 raft_consensus.cc:2229] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:02.838685 29549 raft_consensus.cc:2229] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:02.838770 29550 raft_consensus.cc:2258] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:02.838790 29549 raft_consensus.cc:2258] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:02.839304 29549 ts_tablet_manager.cc:1905] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:02.839955 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 28228
I20260619 02:50:02.841279 29549 ts_tablet_manager.cc:1918] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:02.841341 29549 log.cc:1196] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-0/wal/wals/ffbb67e09c8a40b9a6ca13f46991e638
I20260619 02:50:02.841563 29549 ts_tablet_manager.cc:1939] T ffbb67e09c8a40b9a6ca13f46991e638 P 33bc48e3da76477e9a8136915ad87b63: Deleting consensus metadata
I20260619 02:50:02.844833 29552 tablet_replica.cc:330] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06: stopping tablet replica
I20260619 02:50:02.845204 29552 raft_consensus.cc:2229] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:50:02.845369 29552 raft_consensus.cc:2258] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:50:02.845788 29111 connection.cc:535] server connection from 127.0.0.1:50130 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:50:02.845894 29552 ts_tablet_manager.cc:1905] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:02.846052 29550 ts_tablet_manager.cc:1905] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:02.847496 29552 ts_tablet_manager.cc:1918] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:02.847568 29552 log.cc:1196] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/ffbb67e09c8a40b9a6ca13f46991e638
I20260619 02:50:02.847630 29550 ts_tablet_manager.cc:1918] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:02.847707 29550 log.cc:1196] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/ffbb67e09c8a40b9a6ca13f46991e638
I20260619 02:50:02.847832 29552 ts_tablet_manager.cc:1939] T ffbb67e09c8a40b9a6ca13f46991e638 P 1c38c65a824845b4b92ab1da928b5f06: Deleting consensus metadata
I20260619 02:50:02.847976 29550 ts_tablet_manager.cc:1939] T ffbb67e09c8a40b9a6ca13f46991e638 P ca7f70e8e0f340a5b488afc254535a6a: Deleting consensus metadata
W20260619 02:50:02.848184 29386 connection.cc:441] server connection from 127.0.0.1:37744 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37744 (request call id 1) could send its response
W20260619 02:50:02.848320 29248 connection.cc:441] server connection from 127.0.0.1:35150 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35150 (request call id 1) could send its response
I20260619 02:50:02.849563 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.838958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.21.164.252:37545)
0619 02:50:02.838996 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.838998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.839006 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:02.839084 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.839086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.839097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.839124 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.839878 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.840543 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.842200 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.842503 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.842545 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.844149 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.844150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.844150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.844158 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.845736 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.848690 (+ 2954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.848691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.848692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.848760 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.848976 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.848976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.848977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.849109 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.849183 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.849380 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.849467 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:50:02.856415 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.849222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.21.164.253:33753)
0619 02:50:02.849266 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.849267 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.849393 (+ 126us) server_negotiation.cc:407] Connection header received
0619 02:50:02.849485 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.849486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.849500 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.849528 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.850532 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.851123 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.851823 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.852087 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.852154 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.853682 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.853684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.853684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.853691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.855207 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.855612 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.855613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.855613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.855669 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.855855 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.855856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.855856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.855985 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.856043 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.856223 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.856318 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:50:02.856427 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:02.848535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.21.164.252:37545)
0619 02:50:02.848689 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:02.848691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:02.848938 (+ 247us) server_negotiation.cc:407] Connection header received
0619 02:50:02.848958 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:02.848959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:02.848968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:02.849023 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:02.850019 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.850745 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.851967 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:02.852240 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:02.852303 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:02.853561 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:02.853562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:02.853563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:02.853569 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:02.855384 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.855709 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.855710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.855711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.855769 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:02.855932 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:02.855932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:02.855932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:02.856065 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:02.856154 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:02.856237 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 02:50:02.856325 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260619 02:50:03.834491 29560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.834147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.194:48857)
0619 02:50:03.834268 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.834430 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260619 02:50:03.834692 29360 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:41601 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
I20260619 02:50:03.835170 29561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.834906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.193:36481)
0619 02:50:03.835059 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.835126 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
W20260619 02:50:03.835347 29223 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:41601 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
I20260619 02:50:03.835390 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:03.835189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.195:55817)
0619 02:50:03.835288 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:03.835351 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":45,"threads_started":1}
W20260619 02:50:03.835531 29496 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:41601 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
I20260619 02:50:04.338580 29563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c8a057eee6c4376b5c1a06a6154abd0)
I20260619 02:50:04.338580 29564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c8a057eee6c4376b5c1a06a6154abd0)
I20260619 02:50:04.338713 29564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } }
I20260619 02:50:04.338728 29563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } }
I20260619 02:50:04.338972 29563 leader_election.cc:288] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601), 71dc4518362c47f2a5b5374fd3693ce6 (127.21.164.252:37545)
I20260619 02:50:04.339191 29564 leader_election.cc:288] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601), 868b192a035540c49640f0e4caa8e472 (127.21.164.253:33753)
I20260619 02:50:04.339322 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.339110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52620)
0619 02:50:04.339216 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.339275 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
I20260619 02:50:04.339334 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.339109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52622)
0619 02:50:04.339237 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.339294 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":48,"threads_started":1}
W20260619 02:50:04.339435 29030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
W20260619 02:50:04.339624 28844 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
I20260619 02:50:04.339864 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.339777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52652)
0619 02:50:04.339813 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.339838 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 02:50:04.339951 29030 leader_election.cc:334] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601): Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
I20260619 02:50:04.340003 29568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.339912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52664)
0619 02:50:04.339948 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.339978 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 02:50:04.340070 28844 leader_election.cc:334] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601): Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
I20260619 02:50:04.343108 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.339486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.21.164.252:37545)
0619 02:50:04.339715 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.339716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.339723 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:50:04.339737 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.339738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.339745 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.339935 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:04.339962 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.340765 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.342198 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.342763 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.342839 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.342848 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.342985 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.343025 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":61,"threads_started":1}
I20260619 02:50:04.343128 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.339253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:37545 (local address 127.0.0.1:51066)
0619 02:50:04.339390 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.339399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.339593 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.339833 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.339834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.339836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:04.339880 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.339881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.340847 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.340848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.342099 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.342103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.342842 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.342843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.342917 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.342921 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:04.342975 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:50:04.343026 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":51,"threads_started":1}
I20260619 02:50:04.343116 29567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.339126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:33753 (local address 127.0.0.1:60312)
0619 02:50:04.339286 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.339296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:04.339353 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:04.339674 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:04.339729 (+ 55us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:04.339733 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:04.339795 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.339798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.340868 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.340869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.342099 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:04.342103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.342842 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.342843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:04.342917 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.342921 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:04.342975 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:50:04.343020 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":38,"threads_started":1}
I20260619 02:50:04.343138 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.339326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.21.164.253:33753)
0619 02:50:04.339530 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:04.339532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:04.339539 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:50:04.339564 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:04.339565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:04.339575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:04.339856 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:04.339881 (+ 25us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.340783 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.342194 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:04.342762 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:04.342839 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:04.342848 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:04.342985 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 02:50:04.343028 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":30,"threads_started":1}
I20260619 02:50:04.343561 28868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71dc4518362c47f2a5b5374fd3693ce6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "868b192a035540c49640f0e4caa8e472" is_pre_election: true
I20260619 02:50:04.343565 29054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "868b192a035540c49640f0e4caa8e472" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "71dc4518362c47f2a5b5374fd3693ce6" is_pre_election: true
I20260619 02:50:04.343681 28868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71dc4518362c47f2a5b5374fd3693ce6 in term 1.
I20260619 02:50:04.343681 29054 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868b192a035540c49640f0e4caa8e472 in term 1.
I20260619 02:50:04.343840 28844 leader_election.cc:302] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71dc4518362c47f2a5b5374fd3693ce6, 868b192a035540c49640f0e4caa8e472; no voters: 1c8a057eee6c4376b5c1a06a6154abd0
I20260619 02:50:04.343854 29028 leader_election.cc:302] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71dc4518362c47f2a5b5374fd3693ce6, 868b192a035540c49640f0e4caa8e472; no voters: 1c8a057eee6c4376b5c1a06a6154abd0
I20260619 02:50:04.343963 29563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 02:50:04.343973 29564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 02:50:04.343999 29564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c8a057eee6c4376b5c1a06a6154abd0)
I20260619 02:50:04.343999 29563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1c8a057eee6c4376b5c1a06a6154abd0)
I20260619 02:50:04.344028 29564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:50:04.344028 29563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:50:04.344942 29563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } }
I20260619 02:50:04.344950 29564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } }
I20260619 02:50:04.345094 29563 leader_election.cc:288] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 2 election: Requested vote from peers 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601), 71dc4518362c47f2a5b5374fd3693ce6 (127.21.164.252:37545)
I20260619 02:50:04.345104 29564 leader_election.cc:288] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [CANDIDATE]: Term 2 election: Requested vote from peers 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601), 868b192a035540c49640f0e4caa8e472 (127.21.164.253:33753)
I20260619 02:50:04.345337 29567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.345228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52678)
0619 02:50:04.345276 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.345307 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 02:50:04.345355 28868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71dc4518362c47f2a5b5374fd3693ce6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "868b192a035540c49640f0e4caa8e472"
I20260619 02:50:04.345356 29054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "868b192a035540c49640f0e4caa8e472" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "71dc4518362c47f2a5b5374fd3693ce6"
I20260619 02:50:04.345448 28868 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 71dc4518362c47f2a5b5374fd3693ce6 in current term 2: Already voted for candidate 868b192a035540c49640f0e4caa8e472 in this term.
I20260619 02:50:04.345475 29054 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 868b192a035540c49640f0e4caa8e472 in current term 2: Already voted for candidate 71dc4518362c47f2a5b5374fd3693ce6 in this term.
I20260619 02:50:04.345546 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.345240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52688)
0619 02:50:04.345301 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.345524 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:50:04.345789 29567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.345694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52690)
0619 02:50:04.345739 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.345767 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 02:50:04.345865 29030 leader_election.cc:334] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601): Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
I20260619 02:50:04.345911 29030 leader_election.cc:302] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71dc4518362c47f2a5b5374fd3693ce6; no voters: 1c8a057eee6c4376b5c1a06a6154abd0, 868b192a035540c49640f0e4caa8e472
I20260619 02:50:04.345966 29564 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 02:50:04.346138 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.345904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52700)
0619 02:50:04.346092 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.346115 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
W20260619 02:50:04.346210 28844 leader_election.cc:334] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601): Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
I20260619 02:50:04.346251 28844 leader_election.cc:302] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 868b192a035540c49640f0e4caa8e472; no voters: 1c8a057eee6c4376b5c1a06a6154abd0, 71dc4518362c47f2a5b5374fd3693ce6
I20260619 02:50:04.346354 29563 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 02:50:04.835367 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.835068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.194:44937)
0619 02:50:04.835219 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.835260 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20260619 02:50:04.835839 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.835651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.193:39519)
0619 02:50:04.835766 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.835797 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260619 02:50:04.836030 29575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:04.835795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.195:43539)
0619 02:50:04.835958 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:04.835992 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":55,"threads_started":1}
I20260619 02:50:05.836539 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.835884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.194:58619)
0619 02:50:05.836442 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:05.836488 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":81,"threads_started":1}
I20260619 02:50:05.836539 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.836288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.193:57635)
0619 02:50:05.836442 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:05.836488 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":55,"threads_started":1}
I20260619 02:50:05.836760 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.836537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.195:41933)
0619 02:50:05.836693 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:05.836724 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":44,"threads_started":1}
I20260619 02:50:05.993912 29579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:50:05.994012 29579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } }
I20260619 02:50:05.994153 29579 leader_election.cc:288] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601), 71dc4518362c47f2a5b5374fd3693ce6 (127.21.164.252:37545)
I20260619 02:50:05.994516 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.994306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52920)
0619 02:50:05.994436 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:05.994477 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":50,"threads_started":1}
I20260619 02:50:05.994596 29054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "868b192a035540c49640f0e4caa8e472" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "71dc4518362c47f2a5b5374fd3693ce6" is_pre_election: true
I20260619 02:50:05.994715 29054 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868b192a035540c49640f0e4caa8e472 in term 2.
I20260619 02:50:05.994936 28844 leader_election.cc:302] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71dc4518362c47f2a5b5374fd3693ce6, 868b192a035540c49640f0e4caa8e472; no voters:
I20260619 02:50:05.995023 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.994894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52926)
0619 02:50:05.994957 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:05.994991 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:50:05.995046 29579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 02:50:05.995097 29579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:50:05.995114 29579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 2 FOLLOWER]: Advancing to term 3
W20260619 02:50:05.995126 28844 leader_election.cc:334] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601): Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
I20260619 02:50:05.995869 29579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } }
I20260619 02:50:05.996004 29579 leader_election.cc:288] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 3 election: Requested vote from peers 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601), 71dc4518362c47f2a5b5374fd3693ce6 (127.21.164.252:37545)
I20260619 02:50:05.996212 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.996121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52934)
0619 02:50:05.996157 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:05.996188 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 02:50:05.996330 29054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "868b192a035540c49640f0e4caa8e472" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "71dc4518362c47f2a5b5374fd3693ce6"
I20260619 02:50:05.996475 29054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:50:05.996959 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:05.996875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52948)
0619 02:50:05.996904 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:05.996935 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 02:50:05.997035 28844 leader_election.cc:334] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601): Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
I20260619 02:50:05.997344 29054 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868b192a035540c49640f0e4caa8e472 in term 3.
I20260619 02:50:05.997532 28844 leader_election.cc:302] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71dc4518362c47f2a5b5374fd3693ce6, 868b192a035540c49640f0e4caa8e472; no voters: 1c8a057eee6c4376b5c1a06a6154abd0
I20260619 02:50:05.997620 29579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:50:05.997741 29579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [term 3 LEADER]: Becoming Leader. State: Replica: 868b192a035540c49640f0e4caa8e472, State: Running, Role: LEADER
I20260619 02:50:05.997814 29579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } }
I20260619 02:50:05.998167 29582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 868b192a035540c49640f0e4caa8e472. Latest consensus state: current_term: 3 leader_uuid: "868b192a035540c49640f0e4caa8e472" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:05.998260 29582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:05.998450 29584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:05.998855 29584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:50:05.998924 29584 catalog_manager.cc:1259] Loaded cluster ID: c4c10474c1dd4db7a5cc6420f8dc823c
I20260619 02:50:05.998955 29584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:50:05.999114 29584 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:05.999181 29584 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: Loaded TSK: 0
I20260619 02:50:05.999315 29584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:50:06.018769 28858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:50:06.020426 28858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:50:06.021476 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.021387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52966)
0619 02:50:06.021419 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.021449 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260619 02:50:06.021615 29054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [term 3 FOLLOWER]: Refusing update from remote peer 868b192a035540c49640f0e4caa8e472: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 02:50:06.021770 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.021679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.0.0.1:52968)
0619 02:50:06.021718 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.021739 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 02:50:06.021814 29579 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 02:50:06.021955 28844 consensus_peers.cc:487] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 -> Peer 1c8a057eee6c4376b5c1a06a6154abd0 (127.21.164.254:41601): Couldn't send request to peer 1c8a057eee6c4376b5c1a06a6154abd0. Status: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 02:50:06.022434 29586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 868b192a035540c49640f0e4caa8e472. Latest consensus state: current_term: 3 leader_uuid: "868b192a035540c49640f0e4caa8e472" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:06.022526 29586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:06.022962 29579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "868b192a035540c49640f0e4caa8e472" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:06.023059 29579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:06.023046 29586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "868b192a035540c49640f0e4caa8e472" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1c8a057eee6c4376b5c1a06a6154abd0" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41601 } } peers { permanent_uuid: "868b192a035540c49640f0e4caa8e472" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 33753 } attrs { promote: false } } peers { permanent_uuid: "71dc4518362c47f2a5b5374fd3693ce6" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 37545 } attrs { promote: false } } }
I20260619 02:50:06.023114 29586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71dc4518362c47f2a5b5374fd3693ce6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:50:06.023180 29587 mvcc.cc:205] Tried to move back new op lower bound from 7298406015062265856 to 7298406014967607296. Current Snapshot: MvccSnapshot[applied={T|T < 7298406015062265856}]
I20260619 02:50:06.027805 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.024813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:38869 (local address 127.0.0.1:50132)
0619 02:50:06.025000 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.025010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:06.025056 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:06.025432 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:06.025434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:06.025435 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:06.025476 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.025477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.026278 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.026279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.026972 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.026974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.027564 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.027564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.027633 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.027637 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:06.027689 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:50:06.027726 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":61,"threads_started":1}
I20260619 02:50:06.027806 29595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.024879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.21.164.193:38869)
0619 02:50:06.025294 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:06.025296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:06.025303 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:50:06.025324 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:06.025325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:06.025334 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:06.025530 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:06.025562 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.026215 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.027048 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.027493 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.027561 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.027569 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:06.027698 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 02:50:06.027733 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":49,"threads_started":1}
I20260619 02:50:06.028071 29157 tablet_service.cc:1467] Processing CreateTablet for tablet 936f71fa2e6c48bea36b3c08c651fe28 (DEFAULT_TABLE table=test-table [id=d3763a4d8ccd47ec864b4e31aeb42cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:06.028155 29592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.024733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:46603 (local address 127.0.0.1:37760)
0619 02:50:06.024890 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.024901 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:06.024958 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:06.025139 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:06.025140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:06.025142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:06.025196 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.025198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.026085 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.026085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.027242 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.027244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.027942 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.027943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.028004 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.028008 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:06.028049 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 02:50:06.028089 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260619 02:50:06.028172 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.024741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:40493 (local address 127.0.0.1:35160)
0619 02:50:06.024902 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.024920 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:06.024958 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:06.025318 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:06.025319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:06.025321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:06.025354 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.025356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.025969 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.025970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.027295 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:06.027297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.027946 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.027947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:06.028011 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.028015 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:06.028052 (+ 37us) client_negotiation.cc:240] Negotiation successful
0619 02:50:06.028094 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":31,"thread_start_us":63,"threads_started":1}
I20260619 02:50:06.028208 29157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936f71fa2e6c48bea36b3c08c651fe28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:06.028331 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.024783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.21.164.195:46603)
0619 02:50:06.024990 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:06.024993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:06.025001 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:06.025028 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:06.025028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:06.025038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:06.025104 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:06.025261 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.026013 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.027312 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.027826 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.027903 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.027912 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:06.028222 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 02:50:06.028270 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":53,"threads_started":1}
I20260619 02:50:06.028474 29596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.024925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35160 (local address 127.21.164.194:40493)
0619 02:50:06.025135 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:06.025137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:06.025144 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:50:06.025165 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:06.025166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:06.025175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:06.025228 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:06.025415 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.025896 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.027348 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.027843 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.027904 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.027912 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:06.028294 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 02:50:06.028341 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":44,"threads_started":1}
I20260619 02:50:06.028733 29294 tablet_service.cc:1467] Processing CreateTablet for tablet 936f71fa2e6c48bea36b3c08c651fe28 (DEFAULT_TABLE table=test-table [id=d3763a4d8ccd47ec864b4e31aeb42cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:06.028847 29294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936f71fa2e6c48bea36b3c08c651fe28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:06.029394 29597 tablet_bootstrap.cc:492] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63: Bootstrap starting.
I20260619 02:50:06.029699 29597 tablet_bootstrap.cc:654] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:06.030001 29598 tablet_bootstrap.cc:492] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a: Bootstrap starting.
I20260619 02:50:06.030092 29430 tablet_service.cc:1467] Processing CreateTablet for tablet 936f71fa2e6c48bea36b3c08c651fe28 (DEFAULT_TABLE table=test-table [id=d3763a4d8ccd47ec864b4e31aeb42cd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:06.030198 29430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936f71fa2e6c48bea36b3c08c651fe28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:06.030350 29597 tablet_bootstrap.cc:492] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63: No bootstrap required, opened a new log
I20260619 02:50:06.030383 29598 tablet_bootstrap.cc:654] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:06.030395 29597 ts_tablet_manager.cc:1397] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:06.030514 29597 raft_consensus.cc:348] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } }
I20260619 02:50:06.030611 29597 raft_consensus.cc:374] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:06.030628 29597 raft_consensus.cc:729] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33bc48e3da76477e9a8136915ad87b63, State: Initialized, Role: FOLLOWER
I20260619 02:50:06.030673 29597 consensus_queue.cc:260] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } }
I20260619 02:50:06.030831 29597 ts_tablet_manager.cc:1428] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:06.031416 29225 heartbeater.cc:503] Master 127.21.164.253:33753 was elected leader, sending a full tablet report...
I20260619 02:50:06.031538 29602 tablet_bootstrap.cc:492] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06: Bootstrap starting.
I20260619 02:50:06.031769 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.031666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.193:48363)
0619 02:50:06.031705 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.031742 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 02:50:06.031924 29602 tablet_bootstrap.cc:654] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:06.032167 29598 tablet_bootstrap.cc:492] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a: No bootstrap required, opened a new log
I20260619 02:50:06.032224 29598 ts_tablet_manager.cc:1397] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:50:06.032363 29598 raft_consensus.cc:348] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } }
I20260619 02:50:06.032431 29598 raft_consensus.cc:374] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:06.032450 29598 raft_consensus.cc:729] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7f70e8e0f340a5b488afc254535a6a, State: Initialized, Role: FOLLOWER
I20260619 02:50:06.032477 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.032192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.193:60117)
0619 02:50:06.032424 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.032453 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260619 02:50:06.032500 29598 consensus_queue.cc:260] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } }
I20260619 02:50:06.032707 29598 ts_tablet_manager.cc:1428] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:06.032714 29602 tablet_bootstrap.cc:492] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06: No bootstrap required, opened a new log
I20260619 02:50:06.032757 29602 ts_tablet_manager.cc:1397] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:06.032960 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.032840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.194:35899)
0619 02:50:06.032909 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.032933 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 02:50:06.032940 29602 raft_consensus.cc:348] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } }
I20260619 02:50:06.033001 29602 raft_consensus.cc:374] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:06.033020 29602 raft_consensus.cc:729] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c38c65a824845b4b92ab1da928b5f06, State: Initialized, Role: FOLLOWER
I20260619 02:50:06.033082 29602 consensus_queue.cc:260] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } }
I20260619 02:50:06.033243 29602 ts_tablet_manager.cc:1428] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:06.033380 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.033281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.194:40401)
0619 02:50:06.033332 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.033358 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 02:50:06.033576 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.033450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.195:46785)
0619 02:50:06.033522 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.033550 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 02:50:06.033819 29498 heartbeater.cc:503] Master 127.21.164.253:33753 was elected leader, sending a full tablet report...
I20260619 02:50:06.033861 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.033752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.195:53025)
0619 02:50:06.033788 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.033812 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 02:50:06.033859 29362 heartbeater.cc:503] Master 127.21.164.253:33753 was elected leader, sending a full tablet report...
W20260619 02:50:06.107402 29226 tablet.cc:2369] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:50:06.154409 29499 tablet.cc:2369] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:50:06.163647 29605 raft_consensus.cc:482] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:50:06.163759 29605 raft_consensus.cc:504] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } }
I20260619 02:50:06.163934 29605 leader_election.cc:288] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca7f70e8e0f340a5b488afc254535a6a (127.21.164.194:40493), 33bc48e3da76477e9a8136915ad87b63 (127.21.164.193:38869)
I20260619 02:50:06.164216 29314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936f71fa2e6c48bea36b3c08c651fe28" candidate_uuid: "1c38c65a824845b4b92ab1da928b5f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca7f70e8e0f340a5b488afc254535a6a" is_pre_election: true
I20260619 02:50:06.164215 29177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936f71fa2e6c48bea36b3c08c651fe28" candidate_uuid: "1c38c65a824845b4b92ab1da928b5f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33bc48e3da76477e9a8136915ad87b63" is_pre_election: true
I20260619 02:50:06.164331 29177 raft_consensus.cc:2454] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c38c65a824845b4b92ab1da928b5f06 in term 0.
I20260619 02:50:06.164331 29314 raft_consensus.cc:2454] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c38c65a824845b4b92ab1da928b5f06 in term 0.
I20260619 02:50:06.164484 29385 leader_election.cc:302] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c38c65a824845b4b92ab1da928b5f06, ca7f70e8e0f340a5b488afc254535a6a; no voters:
I20260619 02:50:06.164582 29605 raft_consensus.cc:2781] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:50:06.164633 29605 raft_consensus.cc:482] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:50:06.164683 29605 raft_consensus.cc:3037] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:06.165385 29605 raft_consensus.cc:504] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } }
I20260619 02:50:06.165508 29605 leader_election.cc:288] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [CANDIDATE]: Term 1 election: Requested vote from peers ca7f70e8e0f340a5b488afc254535a6a (127.21.164.194:40493), 33bc48e3da76477e9a8136915ad87b63 (127.21.164.193:38869)
I20260619 02:50:06.165665 29177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936f71fa2e6c48bea36b3c08c651fe28" candidate_uuid: "1c38c65a824845b4b92ab1da928b5f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33bc48e3da76477e9a8136915ad87b63"
I20260619 02:50:06.165741 29177 raft_consensus.cc:3037] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:06.165704 29314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936f71fa2e6c48bea36b3c08c651fe28" candidate_uuid: "1c38c65a824845b4b92ab1da928b5f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca7f70e8e0f340a5b488afc254535a6a"
I20260619 02:50:06.165771 29314 raft_consensus.cc:3037] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:06.166343 29177 raft_consensus.cc:2454] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c38c65a824845b4b92ab1da928b5f06 in term 1.
I20260619 02:50:06.166338 29314 raft_consensus.cc:2454] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c38c65a824845b4b92ab1da928b5f06 in term 1.
I20260619 02:50:06.166524 29384 leader_election.cc:302] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c38c65a824845b4b92ab1da928b5f06, 33bc48e3da76477e9a8136915ad87b63; no voters:
I20260619 02:50:06.166651 29605 raft_consensus.cc:2781] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:06.166802 29605 raft_consensus.cc:686] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 1 LEADER]: Becoming Leader. State: Replica: 1c38c65a824845b4b92ab1da928b5f06, State: Running, Role: LEADER
I20260619 02:50:06.166849 29605 consensus_queue.cc:237] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } } peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } }
I20260619 02:50:06.167131 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.167023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.195:57145)
0619 02:50:06.167067 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.167100 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 02:50:06.167583 28857 catalog_manager.cc:5495] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c38c65a824845b4b92ab1da928b5f06 (127.21.164.195). New cstate: current_term: 1 leader_uuid: "1c38c65a824845b4b92ab1da928b5f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c38c65a824845b4b92ab1da928b5f06" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 46603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 } health_report { overall_health: UNKNOWN } } }
I20260619 02:50:06.207973 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.205986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.21.164.195:46603)
0619 02:50:06.206034 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:06.206036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:06.206043 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:50:06.206088 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:06.206089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:06.206099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:06.206141 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:06.206236 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.206715 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.207472 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:06.207665 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:06.207725 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:06.207803 (+ 78us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:06.207828 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:06.207859 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:06.207895 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 02:50:06.207912 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:50:06.208880 29177 raft_consensus.cc:1261] T 936f71fa2e6c48bea36b3c08c651fe28 P 33bc48e3da76477e9a8136915ad87b63 [term 1 FOLLOWER]: Refusing update from remote peer 1c38c65a824845b4b92ab1da928b5f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:06.208909 29314 raft_consensus.cc:1261] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a [term 1 FOLLOWER]: Refusing update from remote peer 1c38c65a824845b4b92ab1da928b5f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:06.209097 29605 consensus_queue.cc:1035] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33bc48e3da76477e9a8136915ad87b63" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:06.209219 29606 consensus_queue.cc:1035] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca7f70e8e0f340a5b488afc254535a6a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:06.209671 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.209574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.193:33885)
0619 02:50:06.209607 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.209638 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260619 02:50:06.210737 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.210615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.195:58033)
0619 02:50:06.210678 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.210704 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:50:06.210875 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.210796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.193:54725)
0619 02:50:06.210821 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.210846 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
I20260619 02:50:06.210906 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.210772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.194:35433)
0619 02:50:06.210850 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.210875 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 02:50:06.211691 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.211588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.195:50833)
0619 02:50:06.211646 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.211668 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 02:50:06.212164 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.212081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:41601 (local address 127.21.164.194:45601)
0619 02:50:06.212110 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.212135 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260619 02:50:06.213172 28858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:06.213253 28858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:06.214499 28858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 868b192a035540c49640f0e4caa8e472: Sending DeleteTablet for 3 replicas of tablet 936f71fa2e6c48bea36b3c08c651fe28
I20260619 02:50:06.214816 29430 tablet_service.cc:1514] Processing DeleteTablet for tablet 936f71fa2e6c48bea36b3c08c651fe28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:06 UTC) from {username='oryx', principal='oryx/127.21.164.253@KRBTEST.COM'} at 127.0.0.1:37760
I20260619 02:50:06.214828 29294 tablet_service.cc:1514] Processing DeleteTablet for tablet 936f71fa2e6c48bea36b3c08c651fe28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:06 UTC) from {username='oryx', principal='oryx/127.21.164.253@KRBTEST.COM'} at 127.0.0.1:35160
I20260619 02:50:06.215008 29618 tablet_replica.cc:330] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a: stopping tablet replica
I20260619 02:50:06.215065 29618 raft_consensus.cc:2229] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:06.215065 29157 tablet_service.cc:1514] Processing DeleteTablet for tablet 936f71fa2e6c48bea36b3c08c651fe28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:06 UTC) from {username='oryx', principal='oryx/127.21.164.253@KRBTEST.COM'} at 127.0.0.1:50132
I20260619 02:50:06.215142 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 29005
I20260619 02:50:06.215008 29619 tablet_replica.cc:330] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06: stopping tablet replica
I20260619 02:50:06.215615 29619 raft_consensus.cc:2229] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:50:06.215680 29619 raft_consensus.cc:2258] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:06.215775 29618 raft_consensus.cc:2258] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:06.216070 29619 ts_tablet_manager.cc:1905] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:06.217428 29618 ts_tablet_manager.cc:1905] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:06.217622 29619 ts_tablet_manager.cc:1918] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:06.217666 29619 log.cc:1196] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/936f71fa2e6c48bea36b3c08c651fe28
I20260619 02:50:06.217902 29619 ts_tablet_manager.cc:1939] T 936f71fa2e6c48bea36b3c08c651fe28 P 1c38c65a824845b4b92ab1da928b5f06: Deleting consensus metadata
I20260619 02:50:06.218212 28843 catalog_manager.cc:4841] TS 1c38c65a824845b4b92ab1da928b5f06 (127.21.164.195:46603): tablet 936f71fa2e6c48bea36b3c08c651fe28 (table test-table [id=d3763a4d8ccd47ec864b4e31aeb42cd6]) successfully deleted
I20260619 02:50:06.218781 29618 ts_tablet_manager.cc:1918] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:06.218815 29618 log.cc:1196] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/936f71fa2e6c48bea36b3c08c651fe28
I20260619 02:50:06.218969 29618 ts_tablet_manager.cc:1939] T 936f71fa2e6c48bea36b3c08c651fe28 P ca7f70e8e0f340a5b488afc254535a6a: Deleting consensus metadata
I20260619 02:50:06.219440 28844 catalog_manager.cc:4841] TS ca7f70e8e0f340a5b488afc254535a6a (127.21.164.194:40493): tablet 936f71fa2e6c48bea36b3c08c651fe28 (table test-table [id=d3763a4d8ccd47ec864b4e31aeb42cd6]) successfully deleted
I20260619 02:50:06.220492 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:06.220382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:38869 (local address 127.0.0.1:50144)
0619 02:50:06.220431 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:06.220460 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:38869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 02:50:06.220670 28843 catalog_manager.cc:4584] TS 33bc48e3da76477e9a8136915ad87b63 (127.21.164.193:38869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 936f71fa2e6c48bea36b3c08c651fe28: Network error: Client connection negotiation failed: client connection to 127.21.164.193:38869: connect: Connection refused (error 111)
I20260619 02:50:06.220836 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 29231
I20260619 02:50:06.225994 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 29367
I20260619 02:50:06.230608 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 28724
I20260619 02:50:06.234385 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 28912
2026-06-19T02:50:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6232 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:50:06 dist-test-slave-jt0t krb5kdc[29623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:50:06 dist-test-slave-jt0t krb5kdc[29623](info): set up 2 sockets
Jun 19 02:50:06 dist-test-slave-jt0t krb5kdc[29623](info): commencing operation
krb5kdc: starting...
W20260619 02:50:07.188832 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.254.keytab.
Entry for principal oryx/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.254.keytab.
I20260619 02:50:07.311424 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:39121
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.254.keytab
--principal=oryx/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:38259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:07.391817 29636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:07.392040 29636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:07.392061 29636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:07.393222 29636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:07.393257 29636 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:50:07.393270 29636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:07.393281 29636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:07.393291 29636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:07.393302 29636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:07.394778 29636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:38259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:39121
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.254.keytab
--principal=oryx/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:07.394997 29636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:07.395179 29636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:07.400043 29641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:07.400107 29636 server_base.cc:1034] running on GCE node
W20260619 02:50:07.400120 29642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:07.400043 29644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:07.400542 29636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:07.400840 29636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:07.401991 29636 hybrid_clock.cc:648] HybridClock initialized: now 1781837407401978 us; error 36 us; skew 500 ppm
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:07.404507 29636 init.cc:377] Logged in from keytab as oryx/127.21.164.254@KRBTEST.COM (short username oryx)
I20260619 02:50:07.404963 29636 webserver.cc:460] Webserver started at http://127.21.164.254:42857/ using document root <none> and password file <none>
I20260619 02:50:07.405145 29636 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:07.405184 29636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:07.405278 29636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:07.406282 29636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "1c98b90a2a8048fda8fce23d3f74c9dd"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "d26379c88c68afad9f49094b1e04fea4"
server_key_iv: "b0acb28799db310929036e31d7dbc8e7"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.406589 29636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "1c98b90a2a8048fda8fce23d3f74c9dd"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "d26379c88c68afad9f49094b1e04fea4"
server_key_iv: "b0acb28799db310929036e31d7dbc8e7"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.407752 29636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:50:07.408468 29651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:07.408625 29636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:50:07.408735 29636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "1c98b90a2a8048fda8fce23d3f74c9dd"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "d26379c88c68afad9f49094b1e04fea4"
server_key_iv: "b0acb28799db310929036e31d7dbc8e7"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.408806 29636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:07.425009 29636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:07.427102 29636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:07.427234 29636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:07.431604 29636 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:39121
I20260619 02:50:07.431675 29713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:39121 every 8 connection(s)
I20260619 02:50:07.431990 29636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:50:07.432612 29714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:07.434854 29714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd: Bootstrap starting.
I20260619 02:50:07.435425 29714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:07.435693 29714 log.cc:824] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:07.436177 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 29636
I20260619 02:50:07.436276 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:50:07.436437 29714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd: No bootstrap required, opened a new log
I20260619 02:50:07.436482 22163 external_mini_cluster.cc:1428] Setting key f84953e2a6428587b5632361342ed48e
I20260619 02:50:07.438602 29714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c98b90a2a8048fda8fce23d3f74c9dd" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39121 } }
I20260619 02:50:07.438736 29714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:07.438756 29714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c98b90a2a8048fda8fce23d3f74c9dd, State: Initialized, Role: FOLLOWER
I20260619 02:50:07.438869 29714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c98b90a2a8048fda8fce23d3f74c9dd" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39121 } }
I20260619 02:50:07.438943 29714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:50:07.438978 29714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:50:07.439024 29714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:07.439522 29714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c98b90a2a8048fda8fce23d3f74c9dd" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39121 } }
I20260619 02:50:07.439605 29714 leader_election.cc:302] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c98b90a2a8048fda8fce23d3f74c9dd; no voters:
I20260619 02:50:07.439755 29714 leader_election.cc:288] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:50:07.439803 29719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:07.439949 29719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [term 1 LEADER]: Becoming Leader. State: Replica: 1c98b90a2a8048fda8fce23d3f74c9dd, State: Running, Role: LEADER
I20260619 02:50:07.440012 29714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:07.440124 29719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c98b90a2a8048fda8fce23d3f74c9dd" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39121 } }
I20260619 02:50:07.440501 29719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c98b90a2a8048fda8fce23d3f74c9dd. Latest consensus state: current_term: 1 leader_uuid: "1c98b90a2a8048fda8fce23d3f74c9dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c98b90a2a8048fda8fce23d3f74c9dd" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39121 } } }
I20260619 02:50:07.440510 29721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c98b90a2a8048fda8fce23d3f74c9dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c98b90a2a8048fda8fce23d3f74c9dd" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39121 } } }
I20260619 02:50:07.440725 29721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:07.440713 29719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd [sys.catalog]: This master's current role is: LEADER
W20260619 02:50:07.441761 29734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:50:07.441869 29734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:50:07.441936 29728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:07.442078 29728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:50:07.444031 29728 catalog_manager.cc:1347] Generated new cluster ID: 34da3a28cf4a42b08ade899c6a5f7901
I20260619 02:50:07.444074 29728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:07.448889 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.437037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.21.164.254:39121)
0619 02:50:07.437220 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.437228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.437253 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:50:07.438588 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.438597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.438653 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.438793 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.439475 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.440749 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.441848 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.442450 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.442559 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.445188 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:07.445192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:07.445192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:07.445203 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:07.447124 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.447544 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.447546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.447547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.447661 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.447848 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.447849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.447849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.448150 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:07.448309 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:07.448409 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 02:50:07.448675 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260619 02:50:07.462035 29728 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:50:07.462775 29728 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:07.468760 29728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd: Generated new TSK 0
I20260619 02:50:07.469005 29728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab.
Entry for principal oryx/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab.
I20260619 02:50:07.499778 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39121
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab
--principal=oryx/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:38259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:07.580189 29740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:07.580423 29740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:07.580456 29740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:07.581697 29740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:07.581745 29740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:07.581775 29740 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:50:07.581799 29740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:50:07.583375 29740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:38259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab
--principal=oryx/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.164.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:07.583628 29740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:07.583848 29740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:07.588665 29746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:07.588675 29745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:07.588773 29740 server_base.cc:1034] running on GCE node
W20260619 02:50:07.588626 29748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:07.589080 29740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:07.589286 29740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:07.590417 29740 hybrid_clock.cc:648] HybridClock initialized: now 1781837407590419 us; error 46 us; skew 500 ppm
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:07.592877 29740 init.cc:377] Logged in from keytab as oryx/127.21.164.193@KRBTEST.COM (short username oryx)
I20260619 02:50:07.593276 29740 webserver.cc:460] Webserver started at http://127.21.164.193:40201/ using document root <none> and password file <none>
I20260619 02:50:07.593472 29740 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:07.593511 29740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:07.593606 29740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:07.594620 29740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "8ebe4aa24e754480b9d842755031c1dc"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "24d21e85ccce9843c740d5b3bd337aaa"
server_key_iv: "89a1cc9266de64ebafeaaaafca5eb3bf"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.594900 29740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "8ebe4aa24e754480b9d842755031c1dc"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "24d21e85ccce9843c740d5b3bd337aaa"
server_key_iv: "89a1cc9266de64ebafeaaaafca5eb3bf"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.596019 29740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:50:07.596685 29755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:07.596825 29740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:07.596901 29740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "8ebe4aa24e754480b9d842755031c1dc"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "24d21e85ccce9843c740d5b3bd337aaa"
server_key_iv: "89a1cc9266de64ebafeaaaafca5eb3bf"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.596968 29740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:07.606148 29740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:07.608170 29740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:07.608297 29740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:07.608856 29740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:07.608894 29740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:07.608916 29740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:07.608932 29740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:07.615149 29740 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:35077
I20260619 02:50:07.615203 29870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:35077 every 8 connection(s)
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:07.615548 29740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:50:07.618105 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.609235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:43463 (local address 127.21.164.254:39121)
0619 02:50:07.609293 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.609296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.609461 (+ 165us) server_negotiation.cc:407] Connection header received
0619 02:50:07.609929 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.609930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.609942 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.610016 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.610757 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.611563 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.613093 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.613403 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.613481 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.615916 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:07.615917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:07.615918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:07.615926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:07.617055 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.617417 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.617417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.617417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.617464 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.617626 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.617626 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.617627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.617725 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:07.617772 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:07.617975 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 02:50:07.618040 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:50:07.618268 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.609205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39121 (local address 127.21.164.193:43463)
0619 02:50:07.609343 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.609381 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.609834 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.610090 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.610094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.610380 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:07.610655 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.610661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.611657 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.611658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.613003 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.613006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.613482 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.613482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.613592 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.614105 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:07.614115 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:07.615800 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:07.617121 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.617127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.617129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.617334 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.617504 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.617505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.617505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.617586 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.617768 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:07.617771 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:07.617880 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 02:50:07.617962 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:50:07.618079 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260619 02:50:07.618709 29871 heartbeater.cc:347] Connected to a master server at 127.21.164.254:39121
I20260619 02:50:07.618790 29871 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:07.618959 29871 heartbeater.cc:511] Master 127.21.164.254:39121 requested a full tablet report, sending...
I20260619 02:50:07.619539 29667 ts_manager.cc:194] Registered new tserver with Master: 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077)
I20260619 02:50:07.620393 29667 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:43463
I20260619 02:50:07.624542 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.619713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:36171 (local address 127.21.164.254:39121)
0619 02:50:07.619753 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.619755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.619800 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:50:07.619865 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.619866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.619873 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.619924 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.620232 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.620865 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.621529 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.621710 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.621763 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.622604 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:07.622605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:07.622605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:07.622611 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:07.623706 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.624008 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.624008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.624009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.624052 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.624178 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.624179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.624179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.624271 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:07.624315 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:07.624412 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 02:50:07.624477 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:50:07.624585 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.619621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39121 (local address 127.21.164.193:36171)
0619 02:50:07.619745 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.619756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.619814 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.619936 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.619936 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.620125 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:07.620179 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.620181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.620960 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.620961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.621460 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.621461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.621765 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.621765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.621806 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.622227 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:07.622231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:07.622535 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:07.623785 (+ 1250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.623787 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.623787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.623953 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.624096 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.624096 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.624097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.624141 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.624310 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:07.624311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:07.624348 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 02:50:07.624389 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 02:50:07.624491 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260619 02:50:07.624759 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 29740
I20260619 02:50:07.624861 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:50:07.625061 22163 external_mini_cluster.cc:1428] Setting key 0ef834afe6e4b269ed6aff9997195080
WARNING: no policy specified for oryx/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab.
Entry for principal oryx/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab.
I20260619 02:50:07.647261 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39121
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab
--principal=oryx/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:38259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:07.727054 29880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:07.727283 29880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:07.727308 29880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:07.728518 29880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:07.728569 29880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:07.728588 29880 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:50:07.728605 29880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:50:07.730195 29880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:38259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab
--principal=oryx/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.164.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:07.730444 29880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:07.730669 29880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:07.735438 29886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:07.735459 29888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:07.735459 29885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:07.735628 29880 server_base.cc:1034] running on GCE node
I20260619 02:50:07.735942 29880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:07.736195 29880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:07.737340 29880 hybrid_clock.cc:648] HybridClock initialized: now 1781837407737318 us; error 36 us; skew 500 ppm
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:07.739735 29880 init.cc:377] Logged in from keytab as oryx/127.21.164.194@KRBTEST.COM (short username oryx)
I20260619 02:50:07.740242 29880 webserver.cc:460] Webserver started at http://127.21.164.194:39273/ using document root <none> and password file <none>
I20260619 02:50:07.740442 29880 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:07.740497 29880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:07.740643 29880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:07.741861 29880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "804547446b9a44a9997e8af4fd9e104a"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "0376c5260005aba064b157b6fdd75d24"
server_key_iv: "d9b703151fa6028305a1d55cd6923752"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.742197 29880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "804547446b9a44a9997e8af4fd9e104a"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "0376c5260005aba064b157b6fdd75d24"
server_key_iv: "d9b703151fa6028305a1d55cd6923752"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.743438 29880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:50:07.744097 29895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:07.744246 29880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:07.744328 29880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "804547446b9a44a9997e8af4fd9e104a"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "0376c5260005aba064b157b6fdd75d24"
server_key_iv: "d9b703151fa6028305a1d55cd6923752"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.744395 29880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:07.757524 29880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:07.759703 29880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:07.759847 29880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:07.760337 29880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:07.760381 29880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:07.760411 29880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:07.760427 29880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:07.766654 29880 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:46097
I20260619 02:50:07.766712 30010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:46097 every 8 connection(s)
I20260619 02:50:07.767061 29880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:07.769971 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.760801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:36049 (local address 127.21.164.254:39121)
0619 02:50:07.760841 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.760843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.761088 (+ 245us) server_negotiation.cc:407] Connection header received
0619 02:50:07.761629 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.761631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.761643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.761701 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.762555 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.763310 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.764392 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.764592 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.764673 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.767630 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:07.767632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:07.767633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:07.767640 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:07.768754 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.769103 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.769105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.769105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.769165 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.769370 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.769371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.769371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.769492 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:07.769556 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:07.769801 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 02:50:07.769882 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:50:07.770164 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.760817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39121 (local address 127.21.164.194:36049)
0619 02:50:07.760969 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.761007 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.761532 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.761778 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.761782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.762127 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:07.762441 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.762450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.763433 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.763434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.764307 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.764309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.764701 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.764703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.764841 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.765885 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:07.765897 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:07.767533 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:07.768816 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.768823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.768825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.769038 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.769214 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.769215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.769215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.769317 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.769547 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:07.769553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:07.769686 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 02:50:07.769773 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 02:50:07.769943 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260619 02:50:07.770608 30011 heartbeater.cc:347] Connected to a master server at 127.21.164.254:39121
I20260619 02:50:07.770690 30011 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:07.770859 30011 heartbeater.cc:511] Master 127.21.164.254:39121 requested a full tablet report, sending...
I20260619 02:50:07.771234 29668 ts_manager.cc:194] Registered new tserver with Master: 804547446b9a44a9997e8af4fd9e104a (127.21.164.194:46097)
I20260619 02:50:07.771653 29668 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:36049
I20260619 02:50:07.771828 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 29880
I20260619 02:50:07.771948 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:50:07.772169 22163 external_mini_cluster.cc:1428] Setting key 295cef0c2a2f818a4e9b7d9cd7fd770e
I20260619 02:50:07.776681 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.771599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:51457 (local address 127.21.164.254:39121)
0619 02:50:07.771659 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.771660 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.771697 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:50:07.771755 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.771756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.771765 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.771822 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.772140 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.772788 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.773464 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.773663 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.773717 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.774530 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:07.774531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:07.774531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:07.774537 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:07.775660 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.775948 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.775949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.775949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.776002 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.776160 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.776161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.776161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.776299 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:07.776354 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:07.776497 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 02:50:07.776561 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:50:07.776697 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.771538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39121 (local address 127.21.164.194:51457)
0619 02:50:07.771644 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.771654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.771710 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.771825 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.771826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.772013 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:07.772078 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.772080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.772868 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.772869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.773397 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.773398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.773720 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.773721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.773781 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.774177 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:07.774182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:07.774450 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:07.775729 (+ 1279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.775729 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.775730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.775889 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.776053 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.776054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.776054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.776111 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.776356 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:07.776357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:07.776404 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 02:50:07.776463 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:50:07.776561 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for oryx/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab.
Entry for principal oryx/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab.
I20260619 02:50:07.795986 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39121
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab
--principal=oryx/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:38259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:07.875239 30020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:07.875450 30020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:07.875473 30020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:07.876916 30020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:07.876960 30020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:07.876981 30020 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:50:07.876996 30020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:50:07.878463 30020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:38259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab
--principal=oryx/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.164.254:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:07.878652 30020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:07.878825 30020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:07.883677 30028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:07.883666 30026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:07.883668 30025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:07.883833 30020 server_base.cc:1034] running on GCE node
I20260619 02:50:07.884130 30020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:07.884377 30020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:07.885551 30020 hybrid_clock.cc:648] HybridClock initialized: now 1781837407885534 us; error 28 us; skew 500 ppm
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:07.887980 30020 init.cc:377] Logged in from keytab as oryx/127.21.164.195@KRBTEST.COM (short username oryx)
I20260619 02:50:07.888386 30020 webserver.cc:460] Webserver started at http://127.21.164.195:33461/ using document root <none> and password file <none>
I20260619 02:50:07.888567 30020 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:07.888605 30020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:07.888734 30020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:07.889791 30020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "bb9f9585b3b1420aba7ded324d7539aa"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "17b643e37eb5182272a7eb7e8095ffc0"
server_key_iv: "bdc70558581e0f796e01a5c96f9b0e1a"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.890086 30020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "bb9f9585b3b1420aba7ded324d7539aa"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "17b643e37eb5182272a7eb7e8095ffc0"
server_key_iv: "bdc70558581e0f796e01a5c96f9b0e1a"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.891340 30020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:50:07.891978 30035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:07.892131 30020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:07.892213 30020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "bb9f9585b3b1420aba7ded324d7539aa"
format_stamp: "Formatted at 2026-06-19 02:50:07 on dist-test-slave-jt0t"
server_key: "17b643e37eb5182272a7eb7e8095ffc0"
server_key_iv: "bdc70558581e0f796e01a5c96f9b0e1a"
server_key_version: "encryptionkey@0"
I20260619 02:50:07.892269 30020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:07.900827 30020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:07.902797 30020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:07.902904 30020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:07.903353 30020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:07.903404 30020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:07.903426 30020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:07.903437 30020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:07.909467 30020 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:43491
I20260619 02:50:07.909528 30150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:43491 every 8 connection(s)
I20260619 02:50:07.909848 30020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:07.910586 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 30020
I20260619 02:50:07.910693 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:50:07.910878 22163 external_mini_cluster.cc:1428] Setting key 3d9c69c9549f3208588dc154aabfd5ea
I20260619 02:50:07.913245 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.903814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:37573 (local address 127.21.164.254:39121)
0619 02:50:07.903877 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.903879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.904034 (+ 155us) server_negotiation.cc:407] Connection header received
0619 02:50:07.904562 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.904563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.904577 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.904677 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.905480 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.906223 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.907647 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.907943 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.908021 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.910430 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:07.910431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:07.910431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:07.910438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:07.911925 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.912396 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.912397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.912397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.912461 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.912764 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.912765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.912765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.912874 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:07.912927 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:07.913123 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 02:50:07.913183 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 02:50:07.913403 30044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.903769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39121 (local address 127.21.164.195:37573)
0619 02:50:07.903911 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.903952 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.904454 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.904752 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.904756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.905101 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:07.905367 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.905374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.906308 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.906309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.907562 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.907565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.908031 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.908032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.908138 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.908642 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:07.908702 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:07.910346 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:07.912021 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.912027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.912029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.912331 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.912546 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.912547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.912547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.912694 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.912916 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:07.912919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:07.913031 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 02:50:07.913102 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 02:50:07.913221 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260619 02:50:07.913839 30151 heartbeater.cc:347] Connected to a master server at 127.21.164.254:39121
I20260619 02:50:07.913924 30151 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:07.914088 30151 heartbeater.cc:511] Master 127.21.164.254:39121 requested a full tablet report, sending...
I20260619 02:50:07.914424 29667 ts_manager.cc:194] Registered new tserver with Master: bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491)
I20260619 02:50:07.914872 29667 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:37573
I20260619 02:50:07.919435 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.914783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:49907 (local address 127.21.164.254:39121)
0619 02:50:07.914827 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.914829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.914903 (+ 74us) server_negotiation.cc:407] Connection header received
0619 02:50:07.914945 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.914946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.914954 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.915009 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.915334 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.915886 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.916553 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.916769 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.916814 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.917586 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:07.917587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:07.917587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:07.917593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:07.918646 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.918929 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.918930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.918930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.918976 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.919089 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.919089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.919090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.919184 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:07.919230 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:07.919317 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 02:50:07.919372 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:50:07.919445 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.914728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39121 (local address 127.21.164.195:49907)
0619 02:50:07.914850 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.914860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.914909 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.915013 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.915014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.915207 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:07.915264 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.915266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.915967 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.915968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.916461 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.916463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.916813 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.916813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.916860 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.917238 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:07.917242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:07.917508 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:07.918705 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.918706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.918706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.918875 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.919014 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:07.919014 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:07.919015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:07.919059 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:07.919226 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:07.919226 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:07.919259 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:50:07.919301 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:50:07.919372 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":114,"threads_started":1}
I20260619 02:50:07.922382 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:50:07.942212 29718 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:07.936844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.21.164.254:39121)
0619 02:50:07.936915 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.936918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.936926 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:07.936972 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.936973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.936982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.937029 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.937591 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.938079 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.938780 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.938971 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.939024 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.942126 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55758: BlockingRecv error: recv got EOF from 127.0.0.1:55758 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 02:50:07 dist-test-slave-jt0t krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:07.949136 29718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.942885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.21.164.254:39121)
0619 02:50:07.942952 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.942954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.942993 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:50:07.943033 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.943034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.943042 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.943086 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:07.943465 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.943928 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.944594 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.944820 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.944865 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.946824 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:07.946825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:07.946825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:07.946832 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:07.948067 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.948373 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.948373 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.948374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.948448 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:07.948625 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:07.948626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:07.948626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:07.948825 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:07.948908 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:07.948993 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 02:50:07.949051 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:50:07.950665 29667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:50:07.952231 29667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:50:07.958624 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.954474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:35077 (local address 127.0.0.1:40018)
0619 02:50:07.954837 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.954845 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.954879 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.955037 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.955038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.955040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:07.955095 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.955097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.956366 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.956367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.957500 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.957503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.958318 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.958319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.958394 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.958406 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 02:50:07.958484 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:50:07.958538 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":113,"thread_start_us":54,"threads_started":1}
I20260619 02:50:07.958644 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.954717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.21.164.193:35077)
0619 02:50:07.954884 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.954888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.954898 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:07.954935 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.954938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.954952 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.955026 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:07.955182 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.956282 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.957575 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.958232 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.958324 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.958382 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:07.958506 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 02:50:07.958544 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260619 02:50:07.958801 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.954624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.21.164.194:46097)
0619 02:50:07.954791 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.954796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.954806 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:07.954848 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.954850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.954866 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.954955 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:07.955096 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.956325 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.957811 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.958299 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.958385 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.958438 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:07.958665 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 02:50:07.958702 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260619 02:50:07.958963 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.954695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.21.164.195:43491)
0619 02:50:07.954858 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:07.954861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:07.955209 (+ 348us) server_negotiation.cc:407] Connection header received
0619 02:50:07.955252 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:07.955254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:07.955268 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:07.955329 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:07.955521 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.956188 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.957796 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.958515 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.958597 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.958647 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:07.958831 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 02:50:07.958873 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20260619 02:50:07.958916 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.954489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:43491 (local address 127.0.0.1:52618)
0619 02:50:07.955097 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.955106 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.955151 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.955392 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.955392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.955394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:07.955454 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.955455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.956260 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.956261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.957714 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.957716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.958682 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.958683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.958751 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.958756 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:50:07.958802 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:50:07.958843 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":83,"thread_start_us":63,"threads_started":1}
I20260619 02:50:07.958940 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:07.954474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:46097 (local address 127.0.0.1:50322)
0619 02:50:07.954638 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:07.954678 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:07.954742 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:07.954958 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:07.954961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:07.954963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:07.955032 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.955034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.956465 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.956466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.957714 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:07.957716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:07.958470 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:07.958471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:07.958534 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:07.958536 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:50:07.958578 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:50:07.958608 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":50,"thread_start_us":55,"threads_started":1}
I20260619 02:50:07.959759 29805 tablet_service.cc:1467] Processing CreateTablet for tablet 5025217626c34e1582e421d72c815189 (DEFAULT_TABLE table=test-table [id=5896287371f344728bf315a1db9c719a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:07.959841 30085 tablet_service.cc:1467] Processing CreateTablet for tablet 5025217626c34e1582e421d72c815189 (DEFAULT_TABLE table=test-table [id=5896287371f344728bf315a1db9c719a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:07.960148 30085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5025217626c34e1582e421d72c815189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:07.960148 29805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5025217626c34e1582e421d72c815189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:07.961011 29945 tablet_service.cc:1467] Processing CreateTablet for tablet 5025217626c34e1582e421d72c815189 (DEFAULT_TABLE table=test-table [id=5896287371f344728bf315a1db9c719a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:07.961288 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5025217626c34e1582e421d72c815189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:07.963725 30176 tablet_bootstrap.cc:492] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa: Bootstrap starting.
I20260619 02:50:07.964000 30177 tablet_bootstrap.cc:492] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc: Bootstrap starting.
I20260619 02:50:07.964196 30178 tablet_bootstrap.cc:492] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: Bootstrap starting.
I20260619 02:50:07.964506 30176 tablet_bootstrap.cc:654] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:07.964787 30178 tablet_bootstrap.cc:654] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:07.964787 30177 tablet_bootstrap.cc:654] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:07.964820 30176 log.cc:824] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:07.965106 30177 log.cc:824] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:07.965106 30178 log.cc:824] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:07.965600 30176 tablet_bootstrap.cc:492] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa: No bootstrap required, opened a new log
I20260619 02:50:07.965708 30176 ts_tablet_manager.cc:1397] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:50:07.965878 30178 tablet_bootstrap.cc:492] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: No bootstrap required, opened a new log
I20260619 02:50:07.965950 30178 ts_tablet_manager.cc:1397] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:50:07.965994 30177 tablet_bootstrap.cc:492] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc: No bootstrap required, opened a new log
I20260619 02:50:07.966091 30177 ts_tablet_manager.cc:1397] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:50:07.967319 30178 raft_consensus.cc:348] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:07.967458 30178 raft_consensus.cc:374] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:07.967490 30178 raft_consensus.cc:729] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804547446b9a44a9997e8af4fd9e104a, State: Initialized, Role: FOLLOWER
I20260619 02:50:07.967636 30178 consensus_queue.cc:260] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:07.967659 30176 raft_consensus.cc:348] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:07.967828 30176 raft_consensus.cc:374] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:07.967867 30176 raft_consensus.cc:729] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9f9585b3b1420aba7ded324d7539aa, State: Initialized, Role: FOLLOWER
I20260619 02:50:07.967877 30178 ts_tablet_manager.cc:1428] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 02:50:07.968003 30011 heartbeater.cc:503] Master 127.21.164.254:39121 was elected leader, sending a full tablet report...
I20260619 02:50:07.967988 30177 raft_consensus.cc:348] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:07.968134 30177 raft_consensus.cc:374] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:07.968173 30177 raft_consensus.cc:729] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebe4aa24e754480b9d842755031c1dc, State: Initialized, Role: FOLLOWER
I20260619 02:50:07.968272 30177 consensus_queue.cc:260] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:07.968320 30176 consensus_queue.cc:260] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:07.968514 30177 ts_tablet_manager.cc:1428] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 02:50:07.968675 29871 heartbeater.cc:503] Master 127.21.164.254:39121 was elected leader, sending a full tablet report...
I20260619 02:50:07.968706 30151 heartbeater.cc:503] Master 127.21.164.254:39121 was elected leader, sending a full tablet report...
I20260619 02:50:07.968600 30176 ts_tablet_manager.cc:1428] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:50:08.005944 30182 raft_consensus.cc:482] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:50:08.006067 30182 raft_consensus.cc:504] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.006434 30182 leader_election.cc:288] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077), bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491)
I20260619 02:50:08.009202 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.006715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:48885 (local address 127.21.164.195:43491)
0619 02:50:08.006755 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.006757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.006766 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:08.006822 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.006822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.006832 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.006895 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:08.007076 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.007575 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.008494 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.008896 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.008959 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.008968 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.009097 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.009129 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:50:08.009220 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.006682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:34417 (local address 127.21.164.193:35077)
0619 02:50:08.006743 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.006746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.006945 (+ 199us) server_negotiation.cc:407] Connection header received
0619 02:50:08.007015 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.007016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.007026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.007076 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:08.007203 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.007663 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.008533 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.008927 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.008985 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.008993 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.009119 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.009153 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:50:08.009243 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.006624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:43491 (local address 127.21.164.194:48885)
0619 02:50:08.006693 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.006705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.006754 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.006936 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.006937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.006940 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:08.006999 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.007001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.007673 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.007674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.008421 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.008423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.008961 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.008961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.009039 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.009042 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.009085 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.009127 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 02:50:08.009253 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.006625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:35077 (local address 127.21.164.194:34417)
0619 02:50:08.006893 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.006903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.006954 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.007081 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.007082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.007083 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:08.007148 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.007150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.007727 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.007728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.008451 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.008452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.008997 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.008998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.009063 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.009066 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.009109 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.009148 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":95,"thread_start_us":101,"threads_started":1}
I20260619 02:50:08.009603 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025217626c34e1582e421d72c815189" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9f9585b3b1420aba7ded324d7539aa" is_pre_election: true
I20260619 02:50:08.009769 30105 raft_consensus.cc:2454] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 0.
I20260619 02:50:08.009788 29825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025217626c34e1582e421d72c815189" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebe4aa24e754480b9d842755031c1dc" is_pre_election: true
I20260619 02:50:08.009963 29825 raft_consensus.cc:2454] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 0.
I20260619 02:50:08.010008 29897 leader_election.cc:302] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804547446b9a44a9997e8af4fd9e104a, bb9f9585b3b1420aba7ded324d7539aa; no voters:
I20260619 02:50:08.010114 30182 raft_consensus.cc:2781] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:50:08.010169 30182 raft_consensus.cc:482] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:50:08.010190 30182 raft_consensus.cc:3037] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:08.011175 30182 raft_consensus.cc:504] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.011317 30182 leader_election.cc:288] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 election: Requested vote from peers 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077), bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491)
I20260619 02:50:08.011474 29825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025217626c34e1582e421d72c815189" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebe4aa24e754480b9d842755031c1dc"
I20260619 02:50:08.011498 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025217626c34e1582e421d72c815189" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9f9585b3b1420aba7ded324d7539aa"
I20260619 02:50:08.011566 29825 raft_consensus.cc:3037] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:08.011590 30105 raft_consensus.cc:3037] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:08.012384 30105 raft_consensus.cc:2454] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 1.
I20260619 02:50:08.012385 29825 raft_consensus.cc:2454] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 1.
I20260619 02:50:08.012523 29897 leader_election.cc:302] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804547446b9a44a9997e8af4fd9e104a, bb9f9585b3b1420aba7ded324d7539aa; no voters:
I20260619 02:50:08.012605 30182 raft_consensus.cc:2781] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:08.012766 30182 raft_consensus.cc:686] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 1 LEADER]: Becoming Leader. State: Replica: 804547446b9a44a9997e8af4fd9e104a, State: Running, Role: LEADER
I20260619 02:50:08.012856 30182 consensus_queue.cc:237] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.013720 29668 catalog_manager.cc:5495] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a reported cstate change: term changed from 0 to 1, leader changed from <none> to 804547446b9a44a9997e8af4fd9e104a (127.21.164.194). New cstate: current_term: 1 leader_uuid: "804547446b9a44a9997e8af4fd9e104a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } health_report { overall_health: UNKNOWN } } }
W20260619 02:50:08.017580 30012 tablet.cc:2369] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:50:08.031055 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.029008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.21.164.194:46097)
0619 02:50:08.029067 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.029069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.029080 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:50:08.029113 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.029115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.029131 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.029191 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:08.029300 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.029771 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.030455 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.030631 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.030673 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.030778 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:08.030877 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:08.030936 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.030964 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.030980 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:50:08.032793 29825 raft_consensus.cc:1261] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Refusing update from remote peer 804547446b9a44a9997e8af4fd9e104a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:08.032792 30105 raft_consensus.cc:1261] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Refusing update from remote peer 804547446b9a44a9997e8af4fd9e104a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:08.033126 30186 consensus_queue.cc:1035] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:08.033259 30182 consensus_queue.cc:1035] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:08.036172 30190 mvcc.cc:205] Tried to move back new op lower bound from 7298406023299842048 to 7298406023221297152. Current Snapshot: MvccSnapshot[applied={T|T < 7298406023299842048}]
I20260619 02:50:08.036381 30191 mvcc.cc:205] Tried to move back new op lower bound from 7298406023299842048 to 7298406023221297152. Current Snapshot: MvccSnapshot[applied={T|T < 7298406023299842048}]
I20260619 02:50:08.038023 29666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:08.038115 29666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:08.038966 29666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd: Sending DeleteTablet for 3 replicas of tablet 5025217626c34e1582e421d72c815189
I20260619 02:50:08.039289 29805 tablet_service.cc:1514] Processing DeleteTablet for tablet 5025217626c34e1582e421d72c815189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:08 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:40018
I20260619 02:50:08.039286 29945 tablet_service.cc:1514] Processing DeleteTablet for tablet 5025217626c34e1582e421d72c815189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:08 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:50322
I20260619 02:50:08.039526 30198 tablet_replica.cc:330] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc: stopping tablet replica
I20260619 02:50:08.039662 30198 raft_consensus.cc:2229] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:08.039737 30085 tablet_service.cc:1514] Processing DeleteTablet for tablet 5025217626c34e1582e421d72c815189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:08 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:52618
I20260619 02:50:08.039840 30198 raft_consensus.cc:2258] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:08.039955 29666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:50:08.040035 30200 tablet_replica.cc:330] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa: stopping tablet replica
W20260619 02:50:08.040107 29666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:50:08.040125 30200 raft_consensus.cc:2229] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:08.040344 30198 ts_tablet_manager.cc:1905] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:08.040917 30200 raft_consensus.cc:2258] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:08.041422 30200 ts_tablet_manager.cc:1905] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:08.041742 29805 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae2f9a9270347b885d68d3755d88294 (DEFAULT_TABLE table=test-table [id=1b3b1eb634ac4910a829b98329921b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:08.041860 29805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae2f9a9270347b885d68d3755d88294. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:08.042418 30199 tablet_replica.cc:330] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: stopping tablet replica
I20260619 02:50:08.042517 30199 raft_consensus.cc:2229] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:50:08.042620 30199 raft_consensus.cc:2258] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:08.042689 29945 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae2f9a9270347b885d68d3755d88294 (DEFAULT_TABLE table=test-table [id=1b3b1eb634ac4910a829b98329921b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:08.042798 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae2f9a9270347b885d68d3755d88294. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:08.043044 30199 ts_tablet_manager.cc:1905] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:08.043406 30177 tablet_bootstrap.cc:492] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc: Bootstrap starting.
I20260619 02:50:08.043499 30198 ts_tablet_manager.cc:1918] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:08.043562 30198 log.cc:1196] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/wal/wals/5025217626c34e1582e421d72c815189
I20260619 02:50:08.043805 30177 tablet_bootstrap.cc:654] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:08.043820 30198 ts_tablet_manager.cc:1939] T 5025217626c34e1582e421d72c815189 P 8ebe4aa24e754480b9d842755031c1dc: Deleting consensus metadata
I20260619 02:50:08.044126 30178 tablet_bootstrap.cc:492] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a: Bootstrap starting.
I20260619 02:50:08.044512 30178 tablet_bootstrap.cc:654] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:08.044531 30200 ts_tablet_manager.cc:1918] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:08.044596 30200 log.cc:1196] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/5025217626c34e1582e421d72c815189
I20260619 02:50:08.044592 30199 ts_tablet_manager.cc:1918] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:08.044680 30199 log.cc:1196] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/5025217626c34e1582e421d72c815189
I20260619 02:50:08.044864 30200 ts_tablet_manager.cc:1939] T 5025217626c34e1582e421d72c815189 P bb9f9585b3b1420aba7ded324d7539aa: Deleting consensus metadata
I20260619 02:50:08.044919 29652 catalog_manager.cc:4841] TS 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077): tablet 5025217626c34e1582e421d72c815189 (table test-table [id=5896287371f344728bf315a1db9c719a]) successfully deleted
I20260619 02:50:08.044960 30199 ts_tablet_manager.cc:1939] T 5025217626c34e1582e421d72c815189 P 804547446b9a44a9997e8af4fd9e104a: Deleting consensus metadata
I20260619 02:50:08.045369 29655 catalog_manager.cc:4841] TS 804547446b9a44a9997e8af4fd9e104a (127.21.164.194:46097): tablet 5025217626c34e1582e421d72c815189 (table test-table [id=5896287371f344728bf315a1db9c719a]) successfully deleted
I20260619 02:50:08.045467 29653 catalog_manager.cc:4841] TS bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491): tablet 5025217626c34e1582e421d72c815189 (table test-table [id=5896287371f344728bf315a1db9c719a]) successfully deleted
I20260619 02:50:08.045676 30085 tablet_service.cc:1467] Processing CreateTablet for tablet 8ae2f9a9270347b885d68d3755d88294 (DEFAULT_TABLE table=test-table [id=1b3b1eb634ac4910a829b98329921b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:08.045783 30085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ae2f9a9270347b885d68d3755d88294. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:08.046018 30178 tablet_bootstrap.cc:492] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a: No bootstrap required, opened a new log
I20260619 02:50:08.046060 30178 ts_tablet_manager.cc:1397] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:50:08.046200 30178 raft_consensus.cc:348] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.046258 30178 raft_consensus.cc:374] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:08.046275 30178 raft_consensus.cc:729] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804547446b9a44a9997e8af4fd9e104a, State: Initialized, Role: FOLLOWER
I20260619 02:50:08.046326 30178 consensus_queue.cc:260] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.046409 30178 ts_tablet_manager.cc:1428] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.047008 30176 tablet_bootstrap.cc:492] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa: Bootstrap starting.
I20260619 02:50:08.047361 30176 tablet_bootstrap.cc:654] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:08.047902 30176 tablet_bootstrap.cc:492] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa: No bootstrap required, opened a new log
I20260619 02:50:08.047950 30176 ts_tablet_manager.cc:1397] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:08.048108 30176 raft_consensus.cc:348] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.048177 30176 raft_consensus.cc:374] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:08.048197 30176 raft_consensus.cc:729] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9f9585b3b1420aba7ded324d7539aa, State: Initialized, Role: FOLLOWER
I20260619 02:50:08.048236 30176 consensus_queue.cc:260] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.048316 30176 ts_tablet_manager.cc:1428] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:08.054625 30177 tablet_bootstrap.cc:492] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc: No bootstrap required, opened a new log
I20260619 02:50:08.054687 30177 ts_tablet_manager.cc:1397] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20260619 02:50:08.054845 30177 raft_consensus.cc:348] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.054924 30177 raft_consensus.cc:374] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:08.054944 30177 raft_consensus.cc:729] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebe4aa24e754480b9d842755031c1dc, State: Initialized, Role: FOLLOWER
I20260619 02:50:08.054987 30177 consensus_queue.cc:260] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.055080 30177 ts_tablet_manager.cc:1428] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260619 02:50:08.116341 29872 tablet.cc:2369] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:50:08.160403 30152 tablet.cc:2369] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:50:08.449047 30183 raft_consensus.cc:482] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:50:08.449164 30183 raft_consensus.cc:504] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.449544 30183 leader_election.cc:288] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 804547446b9a44a9997e8af4fd9e104a (127.21.164.194:46097), bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491)
I20260619 02:50:08.452504 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.449803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:40989 (local address 127.21.164.194:46097)
0619 02:50:08.449850 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.449853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.449933 (+ 80us) server_negotiation.cc:407] Connection header received
0619 02:50:08.450032 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.450033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.450046 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.450113 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:08.450361 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.450813 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.451830 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.452181 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.452240 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.452248 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.452390 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.452434 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:50:08.452512 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.449918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:57609 (local address 127.21.164.195:43491)
0619 02:50:08.449960 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.449961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.449970 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:08.450014 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.450015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.450024 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.450078 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:08.450283 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.450785 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.451825 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.452169 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.452228 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.452237 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.452388 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.452425 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:50:08.452543 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.449720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:43491 (local address 127.21.164.193:57609)
0619 02:50:08.449903 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.449912 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.449953 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.450085 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.450086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.450088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:08.450173 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.450175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.450884 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.450885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.451729 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.451732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.452241 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.452242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.452321 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.452324 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.452371 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.452425 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":21,"thread_start_us":50,"threads_started":1}
I20260619 02:50:08.452548 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.449718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:46097 (local address 127.21.164.193:40989)
0619 02:50:08.449838 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.449850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.449940 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.450156 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.450157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.450176 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:08.450253 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.450255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.450892 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.450893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.451736 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.451739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.452246 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.452247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.452327 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.452330 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.452375 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.452428 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":44,"thread_start_us":69,"threads_started":1}
I20260619 02:50:08.452929 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae2f9a9270347b885d68d3755d88294" candidate_uuid: "8ebe4aa24e754480b9d842755031c1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804547446b9a44a9997e8af4fd9e104a" is_pre_election: true
I20260619 02:50:08.453061 29965 raft_consensus.cc:2454] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ebe4aa24e754480b9d842755031c1dc in term 0.
I20260619 02:50:08.453040 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae2f9a9270347b885d68d3755d88294" candidate_uuid: "8ebe4aa24e754480b9d842755031c1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9f9585b3b1420aba7ded324d7539aa" is_pre_election: true
I20260619 02:50:08.453117 30105 raft_consensus.cc:2454] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ebe4aa24e754480b9d842755031c1dc in term 0.
I20260619 02:50:08.453258 29759 leader_election.cc:302] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804547446b9a44a9997e8af4fd9e104a, 8ebe4aa24e754480b9d842755031c1dc; no voters:
I20260619 02:50:08.453397 30183 raft_consensus.cc:2781] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:50:08.453446 30183 raft_consensus.cc:482] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:50:08.453466 30183 raft_consensus.cc:3037] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:08.454129 30183 raft_consensus.cc:504] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.454246 30183 leader_election.cc:288] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [CANDIDATE]: Term 1 election: Requested vote from peers 804547446b9a44a9997e8af4fd9e104a (127.21.164.194:46097), bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491)
I20260619 02:50:08.454411 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae2f9a9270347b885d68d3755d88294" candidate_uuid: "8ebe4aa24e754480b9d842755031c1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804547446b9a44a9997e8af4fd9e104a"
I20260619 02:50:08.454433 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ae2f9a9270347b885d68d3755d88294" candidate_uuid: "8ebe4aa24e754480b9d842755031c1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9f9585b3b1420aba7ded324d7539aa"
I20260619 02:50:08.454476 29965 raft_consensus.cc:3037] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:08.454501 30105 raft_consensus.cc:3037] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:08.455098 30105 raft_consensus.cc:2454] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ebe4aa24e754480b9d842755031c1dc in term 1.
I20260619 02:50:08.455139 29965 raft_consensus.cc:2454] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ebe4aa24e754480b9d842755031c1dc in term 1.
I20260619 02:50:08.455274 29757 leader_election.cc:302] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ebe4aa24e754480b9d842755031c1dc, bb9f9585b3b1420aba7ded324d7539aa; no voters:
I20260619 02:50:08.455379 30183 raft_consensus.cc:2781] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:08.455531 30183 raft_consensus.cc:686] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 1 LEADER]: Becoming Leader. State: Replica: 8ebe4aa24e754480b9d842755031c1dc, State: Running, Role: LEADER
I20260619 02:50:08.455603 30183 consensus_queue.cc:237] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.456280 29668 catalog_manager.cc:5495] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193). New cstate: current_term: 1 leader_uuid: "8ebe4aa24e754480b9d842755031c1dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:50:08 dist-test-slave-jt0t krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837407, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.254@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:08.530004 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.521421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39121 (local address 127.0.0.1:55778)
0619 02:50:08.521596 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.521608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.521691 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.521827 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.521827 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.522257 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:08.522317 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.522320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.523241 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.523243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.523831 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.523833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.524287 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.524288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.524341 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.524937 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:08.524942 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:08.526739 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:08.528742 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:08.528771 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:08.528774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:08.529076 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:08.529273 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:08.529274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:08.529274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:08.529342 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:08.529645 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:08.529656 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:08.529739 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.529809 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.529923 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260619 02:50:08.530077 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.521487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.21.164.254:39121)
0619 02:50:08.521604 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.521607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.521667 (+ 60us) server_negotiation.cc:407] Connection header received
0619 02:50:08.521748 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.521749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.521758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.521814 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:08.522398 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.523161 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.523903 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.524166 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.524229 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.526859 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:08.526861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:08.526861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:08.526868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:08.528624 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:08.529143 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:08.529144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:08.529144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:08.529213 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:08.529429 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:08.529430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:08.529430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:08.529577 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:08.529633 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.529849 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.529977 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260619 02:50:08.532058 29668 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:55778:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:50:08.532291 29668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:50:08.533597 29945 tablet_service.cc:1467] Processing CreateTablet for tablet fb6e490311e84f17b30a973eead3162c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa70626c85da412b91e8049269f484db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:50:08.533643 29805 tablet_service.cc:1467] Processing CreateTablet for tablet fb6e490311e84f17b30a973eead3162c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa70626c85da412b91e8049269f484db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:50:08.533639 30085 tablet_service.cc:1467] Processing CreateTablet for tablet fb6e490311e84f17b30a973eead3162c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aa70626c85da412b91e8049269f484db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:50:08.533761 29805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6e490311e84f17b30a973eead3162c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:08.533761 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6e490311e84f17b30a973eead3162c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:08.533771 30085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6e490311e84f17b30a973eead3162c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:08.534837 30176 tablet_bootstrap.cc:492] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: Bootstrap starting.
I20260619 02:50:08.535151 30178 tablet_bootstrap.cc:492] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a: Bootstrap starting.
I20260619 02:50:08.535198 30176 tablet_bootstrap.cc:654] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:08.535187 30177 tablet_bootstrap.cc:492] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc: Bootstrap starting.
I20260619 02:50:08.535452 30178 tablet_bootstrap.cc:654] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:08.535554 30177 tablet_bootstrap.cc:654] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:08.535888 30176 tablet_bootstrap.cc:492] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: No bootstrap required, opened a new log
I20260619 02:50:08.535931 30176 ts_tablet_manager.cc:1397] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:08.536070 30176 raft_consensus.cc:348] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.536125 30177 tablet_bootstrap.cc:492] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc: No bootstrap required, opened a new log
I20260619 02:50:08.536139 30176 raft_consensus.cc:374] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:08.536157 30176 raft_consensus.cc:729] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9f9585b3b1420aba7ded324d7539aa, State: Initialized, Role: FOLLOWER
I20260619 02:50:08.536160 30177 ts_tablet_manager.cc:1397] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:50:08.536206 30176 consensus_queue.cc:260] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.536304 30176 ts_tablet_manager.cc:1428] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.536281 30177 raft_consensus.cc:348] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.536338 30177 raft_consensus.cc:374] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:08.536356 30177 raft_consensus.cc:729] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebe4aa24e754480b9d842755031c1dc, State: Initialized, Role: FOLLOWER
I20260619 02:50:08.536392 30177 consensus_queue.cc:260] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.536391 30184 tablet_replica.cc:439] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } } }
I20260619 02:50:08.536464 30177 ts_tablet_manager.cc:1428] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.536487 30184 tablet_replica.cc:442] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:50:08.536707 30183 tablet_replica.cc:439] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } } }
I20260619 02:50:08.536787 30183 tablet_replica.cc:442] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:50:08.536875 30178 tablet_bootstrap.cc:492] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a: No bootstrap required, opened a new log
I20260619 02:50:08.536921 30178 ts_tablet_manager.cc:1397] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:50:08.537024 30178 raft_consensus.cc:348] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.537092 30178 raft_consensus.cc:374] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:08.537109 30178 raft_consensus.cc:729] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804547446b9a44a9997e8af4fd9e104a, State: Initialized, Role: FOLLOWER
I20260619 02:50:08.537149 30178 consensus_queue.cc:260] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.537240 30178 ts_tablet_manager.cc:1428] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.537261 30186 tablet_replica.cc:439] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } } }
I20260619 02:50:08.537341 30186 tablet_replica.cc:442] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:50:08.693321 30184 raft_consensus.cc:482] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:50:08.693449 30184 raft_consensus.cc:504] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.693751 30184 leader_election.cc:288] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077), 804547446b9a44a9997e8af4fd9e104a (127.21.164.194:46097)
I20260619 02:50:08.697712 30220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.694012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:49749 (local address 127.21.164.193:35077)
0619 02:50:08.694120 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.694122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.694148 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:50:08.694219 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.694220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.694231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.694292 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:08.694439 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.695246 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.696784 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.697358 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.697427 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.697434 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.697595 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.697633 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":48,"threads_started":1}
I20260619 02:50:08.697711 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.694039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:59993 (local address 127.21.164.194:46097)
0619 02:50:08.694105 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.694107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.694126 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:50:08.694220 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.694221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.694230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.694287 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:08.694490 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.695255 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.696778 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.697349 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.697426 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.697434 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.697594 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.697633 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:50:08.697772 30218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.693933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:46097 (local address 127.21.164.195:59993)
0619 02:50:08.694073 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.694084 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.694143 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.694296 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.694297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.694302 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:08.694421 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.694423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.695322 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.695323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.696686 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.696690 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.697431 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.697431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.697526 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.697533 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.697584 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.697649 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":92,"thread_start_us":62,"threads_started":1}
I20260619 02:50:08.697962 30219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.693942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:35077 (local address 127.21.164.195:49749)
0619 02:50:08.694098 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.694107 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.694157 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.694296 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.694297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.694299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:08.694355 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.694357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.695322 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.695323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.696687 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.696690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.697431 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.697431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.697526 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.697534 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.697584 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.697650 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":59,"thread_start_us":61,"threads_started":1}
I20260619 02:50:08.697991 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e490311e84f17b30a973eead3162c" candidate_uuid: "bb9f9585b3b1420aba7ded324d7539aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804547446b9a44a9997e8af4fd9e104a" is_pre_election: true
I20260619 02:50:08.698077 29965 raft_consensus.cc:2454] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb9f9585b3b1420aba7ded324d7539aa in term 0.
I20260619 02:50:08.698252 30039 leader_election.cc:302] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804547446b9a44a9997e8af4fd9e104a, bb9f9585b3b1420aba7ded324d7539aa; no voters:
I20260619 02:50:08.698288 29825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e490311e84f17b30a973eead3162c" candidate_uuid: "bb9f9585b3b1420aba7ded324d7539aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebe4aa24e754480b9d842755031c1dc" is_pre_election: true
I20260619 02:50:08.698357 29825 raft_consensus.cc:2454] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb9f9585b3b1420aba7ded324d7539aa in term 0.
I20260619 02:50:08.698403 30184 raft_consensus.cc:2781] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:50:08.698444 30184 raft_consensus.cc:482] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:50:08.698469 30184 raft_consensus.cc:3037] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:08.699162 30184 raft_consensus.cc:504] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.699283 30184 leader_election.cc:288] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [CANDIDATE]: Term 1 election: Requested vote from peers 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077), 804547446b9a44a9997e8af4fd9e104a (127.21.164.194:46097)
I20260619 02:50:08.699463 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e490311e84f17b30a973eead3162c" candidate_uuid: "bb9f9585b3b1420aba7ded324d7539aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804547446b9a44a9997e8af4fd9e104a"
I20260619 02:50:08.699460 29825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6e490311e84f17b30a973eead3162c" candidate_uuid: "bb9f9585b3b1420aba7ded324d7539aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebe4aa24e754480b9d842755031c1dc"
I20260619 02:50:08.699548 29825 raft_consensus.cc:3037] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:08.699555 29965 raft_consensus.cc:3037] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:08.700135 29825 raft_consensus.cc:2454] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb9f9585b3b1420aba7ded324d7539aa in term 1.
I20260619 02:50:08.700205 29965 raft_consensus.cc:2454] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb9f9585b3b1420aba7ded324d7539aa in term 1.
I20260619 02:50:08.700322 30036 leader_election.cc:302] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ebe4aa24e754480b9d842755031c1dc, bb9f9585b3b1420aba7ded324d7539aa; no voters:
I20260619 02:50:08.700428 30184 raft_consensus.cc:2781] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:08.700541 30184 raft_consensus.cc:686] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [term 1 LEADER]: Becoming Leader. State: Replica: bb9f9585b3b1420aba7ded324d7539aa, State: Running, Role: LEADER
I20260619 02:50:08.700601 30184 consensus_queue.cc:237] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.700961 30221 tablet_replica.cc:439] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: TxnStatusTablet state changed. Reason: New leader bb9f9585b3b1420aba7ded324d7539aa. Latest consensus state: current_term: 1 leader_uuid: "bb9f9585b3b1420aba7ded324d7539aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } } }
I20260619 02:50:08.701030 30221 tablet_replica.cc:442] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:50:08.701181 30223 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:50:08.701285 30223 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:50:08.701349 29666 catalog_manager.cc:5495] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa reported cstate change: term changed from 0 to 1, leader changed from <none> to bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195). New cstate: current_term: 1 leader_uuid: "bb9f9585b3b1420aba7ded324d7539aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } health_report { overall_health: HEALTHY } } }
I20260619 02:50:08.720484 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.717516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:43491 (local address 127.0.0.1:52626)
0619 02:50:08.717593 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.717606 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.717650 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.717958 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.717959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.717960 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:50:08.718008 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.718010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.718945 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.718946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.719557 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.719559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.720000 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.720001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.720051 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.720066 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:50:08.720320 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:50:08.720323 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.720380 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.720423 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 02:50:08.720470 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.717734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.21.164.195:43491)
0619 02:50:08.717797 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.717799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.717808 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:08.717844 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.717845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.717866 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.717938 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:08.718063 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.718875 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.719616 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.719923 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.719984 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.720143 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:08.720268 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:08.720337 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.720354 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.720374 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:50:08.721655 29825 raft_consensus.cc:1261] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Refusing update from remote peer bb9f9585b3b1420aba7ded324d7539aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:08.721714 29965 raft_consensus.cc:1261] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Refusing update from remote peer bb9f9585b3b1420aba7ded324d7539aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:08.721909 30221 consensus_queue.cc:1035] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:08.722036 30184 consensus_queue.cc:1035] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:08.722646 30183 tablet_replica.cc:439] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc: TxnStatusTablet state changed. Reason: New leader bb9f9585b3b1420aba7ded324d7539aa. Latest consensus state: current_term: 1 leader_uuid: "bb9f9585b3b1420aba7ded324d7539aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } } }
I20260619 02:50:08.722663 30186 tablet_replica.cc:439] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a: TxnStatusTablet state changed. Reason: New leader bb9f9585b3b1420aba7ded324d7539aa. Latest consensus state: current_term: 1 leader_uuid: "bb9f9585b3b1420aba7ded324d7539aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } } }
I20260619 02:50:08.722745 30186 tablet_replica.cc:442] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:50:08.722745 30183 tablet_replica.cc:442] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:50:08.723613 30184 tablet_replica.cc:439] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb9f9585b3b1420aba7ded324d7539aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } } }
I20260619 02:50:08.723695 30184 tablet_replica.cc:442] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:50:08.723670 30221 tablet_replica.cc:439] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb9f9585b3b1420aba7ded324d7539aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } } }
I20260619 02:50:08.723740 30221 tablet_replica.cc:442] T fb6e490311e84f17b30a973eead3162c P bb9f9585b3b1420aba7ded324d7539aa: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:50:08.723899 30183 tablet_replica.cc:439] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb9f9585b3b1420aba7ded324d7539aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } } }
I20260619 02:50:08.723979 30183 tablet_replica.cc:442] T fb6e490311e84f17b30a973eead3162c P 8ebe4aa24e754480b9d842755031c1dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:50:08.724573 30186 tablet_replica.cc:439] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb9f9585b3b1420aba7ded324d7539aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } } }
I20260619 02:50:08.724669 30186 tablet_replica.cc:442] T fb6e490311e84f17b30a973eead3162c P 804547446b9a44a9997e8af4fd9e104a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:50:08.725275 30228 mvcc.cc:205] Tried to move back new op lower bound from 7298406026122043392 to 7298406026038370304. Current Snapshot: MvccSnapshot[applied={T|T < 7298406026122043392}]
I20260619 02:50:08.728485 30220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.725695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.21.164.193:35077)
0619 02:50:08.725735 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.725737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.725783 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:50:08.725851 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.725852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.725872 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.725962 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:08.726048 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.726795 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.727584 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.727857 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.727930 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.728102 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:08.728242 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:08.728314 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.728374 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.728404 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:50:08.733332 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.730675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:43491 (local address 127.21.164.193:42519)
0619 02:50:08.730820 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.730831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.730880 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.730991 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.730992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.730993 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:50:08.731049 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.731051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.731904 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.731905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.732477 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.732479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.732953 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.732954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.733027 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.733040 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:50:08.733196 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:50:08.733197 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.733226 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.733259 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260619 02:50:08.733327 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.730748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:42519 (local address 127.21.164.195:43491)
0619 02:50:08.730792 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.730794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.730881 (+ 87us) server_negotiation.cc:407] Connection header received
0619 02:50:08.730919 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.730920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.730929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.730980 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:08.731093 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.731808 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.732550 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.732877 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.732943 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.733114 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:08.733146 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:08.733194 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.733234 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.733252 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:50:08.736084 30105 raft_consensus.cc:1261] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Refusing update from remote peer 8ebe4aa24e754480b9d842755031c1dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:08.736084 29965 raft_consensus.cc:1261] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Refusing update from remote peer 8ebe4aa24e754480b9d842755031c1dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:08.736328 30206 consensus_queue.cc:1035] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:08.736600 30206 consensus_queue.cc:1035] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:08.747731 30220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.742219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:50339 (local address 127.21.164.193:35077)
0619 02:50:08.742258 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.742260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.744851 (+ 2591us) server_negotiation.cc:407] Connection header received
0619 02:50:08.744914 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.744915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.744927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.744973 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:08.745122 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.745584 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.746740 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.747110 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.747189 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.747443 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:08.747478 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:08.747542 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.747624 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.747649 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 02:50:08.747753 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.742065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:35077 (local address 127.21.164.195:50339)
0619 02:50:08.744773 (+ 2708us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:08.744786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:08.744851 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:08.744984 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:08.744985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:08.744998 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:50:08.745067 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.745068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.745655 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.745655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.746669 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:08.746672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.747190 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.747192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:08.747307 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.747326 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:50:08.747568 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:50:08.747570 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:50:08.747606 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 02:50:08.747652 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2668,"thread_start_us":70,"threads_started":1}
I20260619 02:50:08.754990 29668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:08.755092 29668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:08.755625 29668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd: Sending DeleteTablet for 3 replicas of tablet 8ae2f9a9270347b885d68d3755d88294
I20260619 02:50:08.755883 30085 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae2f9a9270347b885d68d3755d88294 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:08 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:52618
I20260619 02:50:08.756105 30247 tablet_replica.cc:330] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa: stopping tablet replica
I20260619 02:50:08.756086 29945 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae2f9a9270347b885d68d3755d88294 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:08 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:50322
I20260619 02:50:08.756245 30247 raft_consensus.cc:2229] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:08.756335 30247 raft_consensus.cc:2258] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:08.756537 29805 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ae2f9a9270347b885d68d3755d88294 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:08 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:40018
I20260619 02:50:08.758922 30249 tablet_replica.cc:330] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc: stopping tablet replica
I20260619 02:50:08.758981 30249 raft_consensus.cc:2229] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:50:08.759083 30249 raft_consensus.cc:2258] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:08.759189 30248 tablet_replica.cc:330] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a: stopping tablet replica
I20260619 02:50:08.759245 30248 raft_consensus.cc:2229] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:08.759287 30248 raft_consensus.cc:2258] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:08.759559 30248 ts_tablet_manager.cc:1905] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:08.760936 30249 ts_tablet_manager.cc:1905] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:08.762136 30249 ts_tablet_manager.cc:1918] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:50:08.762187 30249 log.cc:1196] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/wal/wals/8ae2f9a9270347b885d68d3755d88294
I20260619 02:50:08.762372 30249 ts_tablet_manager.cc:1939] T 8ae2f9a9270347b885d68d3755d88294 P 8ebe4aa24e754480b9d842755031c1dc: Deleting consensus metadata
I20260619 02:50:08.762653 30248 ts_tablet_manager.cc:1918] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:50:08.762707 30248 log.cc:1196] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/8ae2f9a9270347b885d68d3755d88294
I20260619 02:50:08.762703 29652 catalog_manager.cc:4841] TS 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077): tablet 8ae2f9a9270347b885d68d3755d88294 (table test-table [id=1b3b1eb634ac4910a829b98329921b56]) successfully deleted
I20260619 02:50:08.762887 30248 ts_tablet_manager.cc:1939] T 8ae2f9a9270347b885d68d3755d88294 P 804547446b9a44a9997e8af4fd9e104a: Deleting consensus metadata
I20260619 02:50:08.763156 29655 catalog_manager.cc:4841] TS 804547446b9a44a9997e8af4fd9e104a (127.21.164.194:46097): tablet 8ae2f9a9270347b885d68d3755d88294 (table test-table [id=1b3b1eb634ac4910a829b98329921b56]) successfully deleted
I20260619 02:50:08.764823 30247 ts_tablet_manager.cc:1905] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:08.766242 30247 ts_tablet_manager.cc:1918] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:50:08.766302 30247 log.cc:1196] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/8ae2f9a9270347b885d68d3755d88294
I20260619 02:50:08.766489 30247 ts_tablet_manager.cc:1939] T 8ae2f9a9270347b885d68d3755d88294 P bb9f9585b3b1420aba7ded324d7539aa: Deleting consensus metadata
I20260619 02:50:08.769531 29653 catalog_manager.cc:4841] TS bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491): tablet 8ae2f9a9270347b885d68d3755d88294 (table test-table [id=1b3b1eb634ac4910a829b98329921b56]) successfully deleted
Jun 19 02:50:08 dist-test-slave-jt0t krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:50:08 dist-test-slave-jt0t krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:08 dist-test-slave-jt0t krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:50:08.785284 30214 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:08.780338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.21.164.254:39121)
0619 02:50:08.780387 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.780389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.780508 (+ 119us) server_negotiation.cc:407] Connection header received
0619 02:50:08.780572 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.780574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.780585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.780633 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:08.781201 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.781697 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.782408 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.782602 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.782657 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.785231 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55782: BlockingRecv error: recv got EOF from 127.0.0.1:55782 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 19 02:50:08 dist-test-slave-jt0t krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM
I20260619 02:50:08.792914 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:08.786268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.21.164.254:39121)
0619 02:50:08.786307 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:08.786309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:08.786489 (+ 180us) server_negotiation.cc:407] Connection header received
0619 02:50:08.786572 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:08.786574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:08.786584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:08.786628 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:08.787082 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.787604 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.788325 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:08.788535 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:08.788589 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:08.790647 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:08.790649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:08.790650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:08.790657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:08.791903 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:08.792196 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:08.792198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:08.792198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:08.792247 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:08.792403 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:08.792403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:08.792404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:08.792511 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:08.792596 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:08.792732 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 02:50:08.792808 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:50:08.794323 29668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:50:08.794498 29668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:50:08.795766 30085 tablet_service.cc:1467] Processing CreateTablet for tablet 9e408f3af082443a923cefd1de448722 (DEFAULT_TABLE table=test-table [id=5bf68ff57dd04bb6bcc3722e3b8e3faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:08.795766 29805 tablet_service.cc:1467] Processing CreateTablet for tablet 9e408f3af082443a923cefd1de448722 (DEFAULT_TABLE table=test-table [id=5bf68ff57dd04bb6bcc3722e3b8e3faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:08.795892 30085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e408f3af082443a923cefd1de448722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:08.795900 29805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e408f3af082443a923cefd1de448722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:08.797176 30176 tablet_bootstrap.cc:492] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa: Bootstrap starting.
I20260619 02:50:08.797427 30177 tablet_bootstrap.cc:492] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc: Bootstrap starting.
I20260619 02:50:08.797585 30176 tablet_bootstrap.cc:654] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:08.797827 30177 tablet_bootstrap.cc:654] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:08.798357 30176 tablet_bootstrap.cc:492] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa: No bootstrap required, opened a new log
I20260619 02:50:08.798357 29945 tablet_service.cc:1467] Processing CreateTablet for tablet 9e408f3af082443a923cefd1de448722 (DEFAULT_TABLE table=test-table [id=5bf68ff57dd04bb6bcc3722e3b8e3faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:08.798409 30176 ts_tablet_manager.cc:1397] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:08.798462 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e408f3af082443a923cefd1de448722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:08.798547 30176 raft_consensus.cc:348] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.798621 30176 raft_consensus.cc:374] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:08.798638 30176 raft_consensus.cc:729] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9f9585b3b1420aba7ded324d7539aa, State: Initialized, Role: FOLLOWER
I20260619 02:50:08.798676 30176 consensus_queue.cc:260] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.798760 30176 ts_tablet_manager.cc:1428] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:08.799541 30178 tablet_bootstrap.cc:492] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a: Bootstrap starting.
I20260619 02:50:08.799839 30178 tablet_bootstrap.cc:654] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:08.800426 30177 tablet_bootstrap.cc:492] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc: No bootstrap required, opened a new log
I20260619 02:50:08.800472 30177 ts_tablet_manager.cc:1397] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 02:50:08.800622 30177 raft_consensus.cc:348] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.800714 30177 raft_consensus.cc:374] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:08.800733 30177 raft_consensus.cc:729] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebe4aa24e754480b9d842755031c1dc, State: Initialized, Role: FOLLOWER
I20260619 02:50:08.800770 30177 consensus_queue.cc:260] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.800858 30177 ts_tablet_manager.cc:1428] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:08.801257 30178 tablet_bootstrap.cc:492] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a: No bootstrap required, opened a new log
I20260619 02:50:08.801304 30178 ts_tablet_manager.cc:1397] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:50:08.801430 30178 raft_consensus.cc:348] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.801494 30178 raft_consensus.cc:374] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:08.801513 30178 raft_consensus.cc:729] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804547446b9a44a9997e8af4fd9e104a, State: Initialized, Role: FOLLOWER
I20260619 02:50:08.801553 30178 consensus_queue.cc:260] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:08.801631 30178 ts_tablet_manager.cc:1428] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.028373 30186 raft_consensus.cc:482] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:50:09.028476 30186 raft_consensus.cc:504] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.028625 30186 leader_election.cc:288] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077), bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491)
I20260619 02:50:09.028859 29825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e408f3af082443a923cefd1de448722" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebe4aa24e754480b9d842755031c1dc" is_pre_election: true
I20260619 02:50:09.028846 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e408f3af082443a923cefd1de448722" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9f9585b3b1420aba7ded324d7539aa" is_pre_election: true
I20260619 02:50:09.028942 29825 raft_consensus.cc:2454] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 0.
I20260619 02:50:09.028944 30105 raft_consensus.cc:2454] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 0.
I20260619 02:50:09.029089 29896 leader_election.cc:302] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804547446b9a44a9997e8af4fd9e104a, 8ebe4aa24e754480b9d842755031c1dc; no voters:
I20260619 02:50:09.029192 30186 raft_consensus.cc:2781] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:50:09.029233 30186 raft_consensus.cc:482] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:50:09.029256 30186 raft_consensus.cc:3037] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:09.029925 30186 raft_consensus.cc:504] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.030038 30186 leader_election.cc:288] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 election: Requested vote from peers 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077), bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491)
I20260619 02:50:09.030216 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e408f3af082443a923cefd1de448722" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9f9585b3b1420aba7ded324d7539aa"
I20260619 02:50:09.030216 29825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e408f3af082443a923cefd1de448722" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebe4aa24e754480b9d842755031c1dc"
I20260619 02:50:09.030282 30105 raft_consensus.cc:3037] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:09.030285 29825 raft_consensus.cc:3037] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:09.030882 30105 raft_consensus.cc:2454] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 1.
I20260619 02:50:09.030966 29825 raft_consensus.cc:2454] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 1.
I20260619 02:50:09.031035 29897 leader_election.cc:302] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804547446b9a44a9997e8af4fd9e104a, bb9f9585b3b1420aba7ded324d7539aa; no voters:
I20260619 02:50:09.031131 30186 raft_consensus.cc:2781] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:09.031291 30186 raft_consensus.cc:686] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 1 LEADER]: Becoming Leader. State: Replica: 804547446b9a44a9997e8af4fd9e104a, State: Running, Role: LEADER
I20260619 02:50:09.031350 30186 consensus_queue.cc:237] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.031905 29668 catalog_manager.cc:5495] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a reported cstate change: term changed from 0 to 1, leader changed from <none> to 804547446b9a44a9997e8af4fd9e104a (127.21.164.194). New cstate: current_term: 1 leader_uuid: "804547446b9a44a9997e8af4fd9e104a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } health_report { overall_health: UNKNOWN } } }
I20260619 02:50:09.086438 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.084136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.21.164.194:46097)
0619 02:50:09.084202 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:09.084204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:09.084212 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:09.084235 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:09.084236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:09.084243 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:09.084294 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:09.084405 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:09.084979 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:09.085808 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:09.085995 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:09.086049 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:09.086182 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:09.086230 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:09.086272 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:09.086335 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 02:50:09.086357 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:50:09.087278 30105 raft_consensus.cc:1261] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Refusing update from remote peer 804547446b9a44a9997e8af4fd9e104a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:09.087296 29825 raft_consensus.cc:1261] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Refusing update from remote peer 804547446b9a44a9997e8af4fd9e104a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:09.087520 30265 consensus_queue.cc:1035] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:09.087641 30186 consensus_queue.cc:1035] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:09.090188 29668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:09.090281 29668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:09.091012 29668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd: Sending DeleteTablet for 3 replicas of tablet 9e408f3af082443a923cefd1de448722
I20260619 02:50:09.091270 30085 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e408f3af082443a923cefd1de448722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:09 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:52618
I20260619 02:50:09.091344 29945 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e408f3af082443a923cefd1de448722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:09 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:50322
I20260619 02:50:09.091341 29805 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e408f3af082443a923cefd1de448722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:09 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:40018
I20260619 02:50:09.091385 30247 tablet_replica.cc:330] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa: stopping tablet replica
I20260619 02:50:09.091432 30248 tablet_replica.cc:330] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a: stopping tablet replica
I20260619 02:50:09.091434 30249 tablet_replica.cc:330] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc: stopping tablet replica
I20260619 02:50:09.091442 30247 raft_consensus.cc:2229] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:09.091480 30248 raft_consensus.cc:2229] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:50:09.091485 30247 raft_consensus.cc:2258] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:09.091482 30249 raft_consensus.cc:2229] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:09.091526 30249 raft_consensus.cc:2258] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:09.091537 30248 raft_consensus.cc:2258] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:09.091810 30247 ts_tablet_manager.cc:1905] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:09.091836 29668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:50:09.091924 30249 ts_tablet_manager.cc:1905] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:50:09.091974 29668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:50:09.093185 29945 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf9b93a587947599154fef0c0d15131 (DEFAULT_TABLE table=test-table [id=1d0997f566ad4df298701d64fc7fd1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:09.093302 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf9b93a587947599154fef0c0d15131. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:09.093302 30247 ts_tablet_manager.cc:1918] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:09.093348 30247 log.cc:1196] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/9e408f3af082443a923cefd1de448722
I20260619 02:50:09.093436 30249 ts_tablet_manager.cc:1918] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:09.093495 30249 log.cc:1196] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-0/wal/wals/9e408f3af082443a923cefd1de448722
I20260619 02:50:09.093549 30247 ts_tablet_manager.cc:1939] T 9e408f3af082443a923cefd1de448722 P bb9f9585b3b1420aba7ded324d7539aa: Deleting consensus metadata
I20260619 02:50:09.093693 30249 ts_tablet_manager.cc:1939] T 9e408f3af082443a923cefd1de448722 P 8ebe4aa24e754480b9d842755031c1dc: Deleting consensus metadata
I20260619 02:50:09.093847 29805 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf9b93a587947599154fef0c0d15131 (DEFAULT_TABLE table=test-table [id=1d0997f566ad4df298701d64fc7fd1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:09.093936 29805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf9b93a587947599154fef0c0d15131. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:09.094049 29652 catalog_manager.cc:4841] TS 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077): tablet 9e408f3af082443a923cefd1de448722 (table test-table [id=5bf68ff57dd04bb6bcc3722e3b8e3faa]) successfully deleted
I20260619 02:50:09.094504 30178 tablet_bootstrap.cc:492] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a: Bootstrap starting.
I20260619 02:50:09.094879 30178 tablet_bootstrap.cc:654] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:09.094906 30177 tablet_bootstrap.cc:492] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc: Bootstrap starting.
I20260619 02:50:09.095005 29653 catalog_manager.cc:4841] TS bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491): tablet 9e408f3af082443a923cefd1de448722 (table test-table [id=5bf68ff57dd04bb6bcc3722e3b8e3faa]) successfully deleted
I20260619 02:50:09.095104 30085 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf9b93a587947599154fef0c0d15131 (DEFAULT_TABLE table=test-table [id=1d0997f566ad4df298701d64fc7fd1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:09.095192 30085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf9b93a587947599154fef0c0d15131. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:09.095199 30177 tablet_bootstrap.cc:654] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:09.095216 30248 ts_tablet_manager.cc:1905] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:09.096141 30176 tablet_bootstrap.cc:492] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa: Bootstrap starting.
I20260619 02:50:09.096450 30176 tablet_bootstrap.cc:654] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:09.096547 30178 tablet_bootstrap.cc:492] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a: No bootstrap required, opened a new log
I20260619 02:50:09.096597 30178 ts_tablet_manager.cc:1397] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260619 02:50:09.096607 30177 tablet_bootstrap.cc:492] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc: No bootstrap required, opened a new log
I20260619 02:50:09.096670 30177 ts_tablet_manager.cc:1397] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:50:09.096756 30248 ts_tablet_manager.cc:1918] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:09.096819 30248 log.cc:1196] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/9e408f3af082443a923cefd1de448722
I20260619 02:50:09.096793 30177 raft_consensus.cc:348] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.096863 30177 raft_consensus.cc:374] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:09.096884 30177 raft_consensus.cc:729] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebe4aa24e754480b9d842755031c1dc, State: Initialized, Role: FOLLOWER
I20260619 02:50:09.096935 30177 consensus_queue.cc:260] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.097014 30248 ts_tablet_manager.cc:1939] T 9e408f3af082443a923cefd1de448722 P 804547446b9a44a9997e8af4fd9e104a: Deleting consensus metadata
I20260619 02:50:09.097035 30177 ts_tablet_manager.cc:1428] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.096732 30178 raft_consensus.cc:348] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.097224 30178 raft_consensus.cc:374] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:09.097252 30178 raft_consensus.cc:729] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804547446b9a44a9997e8af4fd9e104a, State: Initialized, Role: FOLLOWER
I20260619 02:50:09.097285 29655 catalog_manager.cc:4841] TS 804547446b9a44a9997e8af4fd9e104a (127.21.164.194:46097): tablet 9e408f3af082443a923cefd1de448722 (table test-table [id=5bf68ff57dd04bb6bcc3722e3b8e3faa]) successfully deleted
I20260619 02:50:09.097293 30178 consensus_queue.cc:260] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.097390 30178 ts_tablet_manager.cc:1428] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:09.097756 30176 tablet_bootstrap.cc:492] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa: No bootstrap required, opened a new log
I20260619 02:50:09.097808 30176 ts_tablet_manager.cc:1397] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260619 02:50:09.097913 30176 raft_consensus.cc:348] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.097967 30176 raft_consensus.cc:374] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:09.097985 30176 raft_consensus.cc:729] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9f9585b3b1420aba7ded324d7539aa, State: Initialized, Role: FOLLOWER
I20260619 02:50:09.098026 30176 consensus_queue.cc:260] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.098116 30176 ts_tablet_manager.cc:1428] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:09.155257 30186 raft_consensus.cc:482] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:50:09.155345 30186 raft_consensus.cc:504] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.155485 30186 leader_election.cc:288] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077), bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491)
I20260619 02:50:09.155679 29825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf9b93a587947599154fef0c0d15131" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebe4aa24e754480b9d842755031c1dc" is_pre_election: true
I20260619 02:50:09.155683 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf9b93a587947599154fef0c0d15131" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9f9585b3b1420aba7ded324d7539aa" is_pre_election: true
I20260619 02:50:09.155761 29825 raft_consensus.cc:2454] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 0.
I20260619 02:50:09.155781 30105 raft_consensus.cc:2454] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 0.
I20260619 02:50:09.155910 29896 leader_election.cc:302] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804547446b9a44a9997e8af4fd9e104a, 8ebe4aa24e754480b9d842755031c1dc; no voters:
I20260619 02:50:09.156008 30186 raft_consensus.cc:2781] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:50:09.156041 30186 raft_consensus.cc:482] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:50:09.156054 30186 raft_consensus.cc:3037] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:09.156725 30186 raft_consensus.cc:504] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.156838 30186 leader_election.cc:288] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 election: Requested vote from peers 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077), bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491)
I20260619 02:50:09.156982 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf9b93a587947599154fef0c0d15131" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9f9585b3b1420aba7ded324d7539aa"
I20260619 02:50:09.157058 30105 raft_consensus.cc:3037] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:09.157025 29825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf9b93a587947599154fef0c0d15131" candidate_uuid: "804547446b9a44a9997e8af4fd9e104a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebe4aa24e754480b9d842755031c1dc"
I20260619 02:50:09.157090 29825 raft_consensus.cc:3037] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:09.157642 30105 raft_consensus.cc:2454] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 1.
I20260619 02:50:09.157649 29825 raft_consensus.cc:2454] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 804547446b9a44a9997e8af4fd9e104a in term 1.
I20260619 02:50:09.157807 29897 leader_election.cc:302] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804547446b9a44a9997e8af4fd9e104a, bb9f9585b3b1420aba7ded324d7539aa; no voters:
I20260619 02:50:09.157898 30186 raft_consensus.cc:2781] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:09.157950 30186 raft_consensus.cc:686] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 1 LEADER]: Becoming Leader. State: Replica: 804547446b9a44a9997e8af4fd9e104a, State: Running, Role: LEADER
I20260619 02:50:09.158000 30186 consensus_queue.cc:237] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } }
I20260619 02:50:09.158541 29666 catalog_manager.cc:5495] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a reported cstate change: term changed from 0 to 1, leader changed from <none> to 804547446b9a44a9997e8af4fd9e104a (127.21.164.194). New cstate: current_term: 1 leader_uuid: "804547446b9a44a9997e8af4fd9e104a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "804547446b9a44a9997e8af4fd9e104a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 46097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 } health_report { overall_health: UNKNOWN } } }
I20260619 02:50:09.176393 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.174164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:44565 (local address 127.21.164.195:43491)
0619 02:50:09.174222 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:09.174224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:09.174346 (+ 122us) server_negotiation.cc:407] Connection header received
0619 02:50:09.174394 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:09.174395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:09.174427 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:09.174463 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:09.174567 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:09.175039 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:09.175787 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:09.175982 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:09.176026 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:09.176217 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:09.176241 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:09.176272 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:09.176315 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 02:50:09.176331 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:50:09.176440 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.174099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:43491 (local address 127.21.164.194:44565)
0619 02:50:09.174283 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:09.174294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:09.174348 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:09.174473 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:09.174474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:09.174475 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:50:09.174531 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:09.174533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:09.175124 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:09.175125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:09.175723 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:09.175726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:09.176070 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:09.176071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:09.176135 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:09.176151 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:50:09.176282 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:50:09.176283 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:50:09.176313 (+ 30us) client_negotiation.cc:240] Negotiation successful
0619 02:50:09.176375 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260619 02:50:09.178264 30105 raft_consensus.cc:1261] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Refusing update from remote peer 804547446b9a44a9997e8af4fd9e104a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:09.178268 29825 raft_consensus.cc:1261] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Refusing update from remote peer 804547446b9a44a9997e8af4fd9e104a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:09.178474 30265 consensus_queue.cc:1035] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb9f9585b3b1420aba7ded324d7539aa" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 43491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:09.178587 30186 consensus_queue.cc:1035] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ebe4aa24e754480b9d842755031c1dc" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:09.184686 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.182556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:42475 (local address 127.21.164.194:46097)
0619 02:50:09.182595 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:09.182597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:09.182605 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:09.182672 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:09.182673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:09.182683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:09.182727 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:09.182952 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:09.183461 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:09.184169 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:09.184343 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:09.184386 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:09.184471 (+ 85us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:09.184495 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:09.184520 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:09.184570 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 02:50:09.184587 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:50:09.184717 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.182511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:46097 (local address 127.21.164.195:42475)
0619 02:50:09.182563 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:09.182575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:09.182616 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:09.182814 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:09.182815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:09.182816 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:50:09.182879 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:09.182880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:09.183544 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:09.183545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:09.184105 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:09.184107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:09.184391 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:09.184392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:09.184427 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:09.184432 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:50:09.184533 (+ 101us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:50:09.184534 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 02:50:09.184570 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 02:50:09.184612 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 02:50:09.190899 29666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:09.190995 29666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:09.191607 29666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c98b90a2a8048fda8fce23d3f74c9dd: Sending DeleteTablet for 3 replicas of tablet 3cf9b93a587947599154fef0c0d15131
I20260619 02:50:09.191840 29805 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf9b93a587947599154fef0c0d15131 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:09 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:40018
I20260619 02:50:09.191840 30085 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf9b93a587947599154fef0c0d15131 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:09 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:52618
I20260619 02:50:09.191886 29945 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf9b93a587947599154fef0c0d15131 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:09 UTC) from {username='oryx', principal='oryx/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:50322
I20260619 02:50:09.191933 30249 tablet_replica.cc:330] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc: stopping tablet replica
I20260619 02:50:09.191958 30247 tablet_replica.cc:330] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa: stopping tablet replica
I20260619 02:50:09.191992 30249 raft_consensus.cc:2229] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:09.191994 30248 tablet_replica.cc:330] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a: stopping tablet replica
I20260619 02:50:09.192003 30247 raft_consensus.cc:2229] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:09.192026 30249 raft_consensus.cc:2258] T 3cf9b93a587947599154fef0c0d15131 P 8ebe4aa24e754480b9d842755031c1dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:09.192031 30247 raft_consensus.cc:2258] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:09.192041 30248 raft_consensus.cc:2229] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:50:09.192093 30248 raft_consensus.cc:2258] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:09.192200 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 29740
I20260619 02:50:09.192428 30248 ts_tablet_manager.cc:1905] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:09.192428 30247 ts_tablet_manager.cc:1905] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:09.194208 30248 ts_tablet_manager.cc:1918] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:50:09.194274 30248 log.cc:1196] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/3cf9b93a587947599154fef0c0d15131
I20260619 02:50:09.194276 30247 ts_tablet_manager.cc:1918] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:50:09.194319 30247 log.cc:1196] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/3cf9b93a587947599154fef0c0d15131
I20260619 02:50:09.194478 30248 ts_tablet_manager.cc:1939] T 3cf9b93a587947599154fef0c0d15131 P 804547446b9a44a9997e8af4fd9e104a: Deleting consensus metadata
I20260619 02:50:09.194501 30247 ts_tablet_manager.cc:1939] T 3cf9b93a587947599154fef0c0d15131 P bb9f9585b3b1420aba7ded324d7539aa: Deleting consensus metadata
I20260619 02:50:09.194783 29655 catalog_manager.cc:4841] TS 804547446b9a44a9997e8af4fd9e104a (127.21.164.194:46097): tablet 3cf9b93a587947599154fef0c0d15131 (table test-table [id=1d0997f566ad4df298701d64fc7fd1e2]) successfully deleted
I20260619 02:50:09.194880 29653 catalog_manager.cc:4841] TS bb9f9585b3b1420aba7ded324d7539aa (127.21.164.195:43491): tablet 3cf9b93a587947599154fef0c0d15131 (table test-table [id=1d0997f566ad4df298701d64fc7fd1e2]) successfully deleted
W20260619 02:50:09.198412 29652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:35077 (error 108)
I20260619 02:50:09.198632 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 29880
I20260619 02:50:09.205708 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 30020
I20260619 02:50:09.208182 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:09.199525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:35077 (local address 127.0.0.1:40032)
0619 02:50:09.208073 (+ 8548us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:09.208136 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8518,"thread_start_us":74,"threads_started":1}
W20260619 02:50:09.208343 29652 catalog_manager.cc:4584] TS 8ebe4aa24e754480b9d842755031c1dc (127.21.164.193:35077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3cf9b93a587947599154fef0c0d15131: Network error: Client connection negotiation failed: client connection to 127.21.164.193:35077: connect: Connection refused (error 111)
I20260619 02:50:09.212435 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 29636
2026-06-19T02:50:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2977 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:50:09 dist-test-slave-jt0t krb5kdc[30284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:50:09 dist-test-slave-jt0t krb5kdc[30284](info): set up 2 sockets
Jun 19 02:50:09 dist-test-slave-jt0t krb5kdc[30284](info): commencing operation
krb5kdc: starting...
W20260619 02:50:10.160948 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:50:10 dist-test-slave-jt0t krb5kdc[30284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:50:10.275637 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:46511
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:46511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:10.355044 30297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:10.355276 30297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:10.355298 30297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:10.356514 30297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:10.356561 30297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:10.356575 30297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:10.356586 30297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:10.356597 30297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:10.358150 30297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:46511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:46511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:10.358392 30297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:10.358620 30297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:10.363373 30303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:10.363430 30302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:10.363409 30305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:10.363423 30297 server_base.cc:1034] running on GCE node
I20260619 02:50:10.363864 30297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:10.364149 30297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:10.365286 30297 hybrid_clock.cc:648] HybridClock initialized: now 1781837410365273 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T02:50:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1161 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:50:10 dist-test-slave-jt0t krb5kdc[30310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:50:10 dist-test-slave-jt0t krb5kdc[30310](info): set up 2 sockets
Jun 19 02:50:10 dist-test-slave-jt0t krb5kdc[30310](info): commencing operation
krb5kdc: starting...
W20260619 02:50:11.328987 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:50:11 dist-test-slave-jt0t krb5kdc[30310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:50:11.445084 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34133
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:34133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:11.523783 30323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:11.524003 30323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:11.524031 30323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:11.525174 30323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:11.525218 30323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:11.525236 30323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:11.525252 30323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:11.525266 30323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:11.526728 30323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:34133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:11.526983 30323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:11.527187 30323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:11.531916 30331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:11.531910 30328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:11.531939 30329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:11.532193 30323 server_base.cc:1034] running on GCE node
I20260619 02:50:11.532433 30323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:11.532783 30323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:11.533984 30323 hybrid_clock.cc:648] HybridClock initialized: now 1781837411533972 us; error 33 us; skew 500 ppm
Jun 19 02:50:11 dist-test-slave-jt0t krb5kdc[30310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:11.541339 30323 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:50:11.541805 30323 webserver.cc:460] Webserver started at http://127.21.164.254:42513/ using document root <none> and password file <none>
I20260619 02:50:11.542018 30323 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:11.542068 30323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:11.542193 30323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:11.543259 30323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "dfb7cdb4e12e8a09226e58715fc07313"
server_key_iv: "2a4f25a6834211524873ad3e8e3749ef"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.543589 30323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "dfb7cdb4e12e8a09226e58715fc07313"
server_key_iv: "2a4f25a6834211524873ad3e8e3749ef"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.544953 30323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:50:11.545634 30338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.545785 30323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:11.545872 30323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "dfb7cdb4e12e8a09226e58715fc07313"
server_key_iv: "2a4f25a6834211524873ad3e8e3749ef"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.545946 30323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:11.573316 30323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:11.575393 30323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:11.575541 30323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:11.579222 30323 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:34133
I20260619 02:50:11.579258 30390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:34133 every 8 connection(s)
I20260619 02:50:11.579591 30323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:50:11.579854 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 30323
I20260619 02:50:11.579973 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:50:11.580159 22163 external_mini_cluster.cc:1428] Setting key f59de79ecb04a0230844725b75ea5939
I20260619 02:50:11.580606 30391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:11.583755 30391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922: Bootstrap starting.
I20260619 02:50:11.584378 30391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:11.584710 30391 log.cc:824] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:11.585510 30391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922: No bootstrap required, opened a new log
Jun 19 02:50:11 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:50:11.587445 30391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } }
I20260619 02:50:11.587599 30391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:11.587632 30391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee924ec0d4b4ea5bba3a3238d7ee922, State: Initialized, Role: FOLLOWER
I20260619 02:50:11.587764 30391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } }
I20260619 02:50:11.587836 30391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:50:11.587873 30391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:50:11.587919 30391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:11.588428 30391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } }
I20260619 02:50:11.588522 30391 leader_election.cc:302] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ee924ec0d4b4ea5bba3a3238d7ee922; no voters:
I20260619 02:50:11.588703 30391 leader_election.cc:288] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:50:11.588733 30396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:11.588843 30396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 1 LEADER]: Becoming Leader. State: Replica: 1ee924ec0d4b4ea5bba3a3238d7ee922, State: Running, Role: LEADER
I20260619 02:50:11.588943 30396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } }
I20260619 02:50:11.588976 30391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:11.589545 30398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee924ec0d4b4ea5bba3a3238d7ee922. Latest consensus state: current_term: 1 leader_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } } }
I20260619 02:50:11.589558 30397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } } }
I20260619 02:50:11.589771 30397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:11.589715 30398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:11.590032 30405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:11.590528 30405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:50:11.591125 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.580684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.21.164.254:34133)
0619 02:50:11.580855 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.580865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.580898 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:50:11.581389 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.581399 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.581465 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.581620 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:11.582137 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.583477 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.584258 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.584885 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.584994 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.587182 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:11.587186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:11.587187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:11.587200 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:11.589066 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.589472 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.589474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.589475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.589584 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.589795 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.589796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.589796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.590119 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:11.590266 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.590772 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.590912 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260619 02:50:11.592319 30405 catalog_manager.cc:1347] Generated new cluster ID: c8d806ec61724273bf0f65b34981c8a1
I20260619 02:50:11.592361 30405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:50:11.597587 30405 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:50:11.598102 30405 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:11.605341 30405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922: Generated new TSK 0
I20260619 02:50:11.605544 30405 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:50:11.633344 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34133
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:11.713832 30417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:11.714056 30417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:11.714082 30417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:11.715286 30417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:11.715329 30417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:11.715359 30417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:50:11.716876 30417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:11.717111 30417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:11.717334 30417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:11.722273 30425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:11.722317 30417 server_base.cc:1034] running on GCE node
W20260619 02:50:11.722288 30422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:11.722267 30423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:11.722676 30417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:11.722915 30417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:11.724064 30417 hybrid_clock.cc:648] HybridClock initialized: now 1781837411724051 us; error 30 us; skew 500 ppm
Jun 19 02:50:11 dist-test-slave-jt0t krb5kdc[30310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:11.726517 30417 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:50:11.726963 30417 webserver.cc:460] Webserver started at http://127.21.164.193:33789/ using document root <none> and password file <none>
I20260619 02:50:11.727169 30417 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:11.727216 30417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:11.727340 30417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:11.728428 30417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "c70fe76e31174e92b0fcce6b39fb8113"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "db6c705942efdf87a8caa39a85835a44"
server_key_iv: "3357bb21636b763bc31fdf1e6d3e488f"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.728789 30417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "c70fe76e31174e92b0fcce6b39fb8113"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "db6c705942efdf87a8caa39a85835a44"
server_key_iv: "3357bb21636b763bc31fdf1e6d3e488f"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.730024 30417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:50:11.730716 30432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.730865 30417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:50:11.730943 30417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "c70fe76e31174e92b0fcce6b39fb8113"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "db6c705942efdf87a8caa39a85835a44"
server_key_iv: "3357bb21636b763bc31fdf1e6d3e488f"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.731010 30417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:11.750865 30417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:11.752718 30417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:11.752861 30417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:11.753077 30417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:11.753387 30417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:11.753427 30417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.753458 30417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:11.753475 30417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.758881 30417 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:37551
I20260619 02:50:11.758922 30545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:37551 every 8 connection(s)
I20260619 02:50:11.759222 30417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:50:11 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:50:11.767541 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.759913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:48945 (local address 127.21.164.254:34133)
0619 02:50:11.759972 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.759974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.760159 (+ 185us) server_negotiation.cc:407] Connection header received
0619 02:50:11.760629 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.760630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.760639 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.760728 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:11.761636 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.762130 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.763050 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.763233 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.763279 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.765295 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:11.765296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:11.765297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:11.765303 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:11.766540 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.766846 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.766846 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.766847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.766893 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.767063 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.767064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.767064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.767158 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:11.767201 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.767396 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.767476 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:50:11.767676 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.759872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:48945)
0619 02:50:11.760041 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.760083 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.760552 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.760799 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.760803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.761358 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:11.761563 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.761568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.762194 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.762195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.762985 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.762987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.763293 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.763293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.763392 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.763848 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:11.763857 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:11.765200 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:11.766600 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:11.766604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:11.766606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:11.766796 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:11.766936 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:11.766936 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:11.766937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:11.767023 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:11.767197 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:11.767200 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:11.767307 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.767381 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.767490 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260619 02:50:11.768106 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 30417
I20260619 02:50:11.768115 30546 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34133
I20260619 02:50:11.768195 30546 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:11.768200 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:50:11.768368 22163 external_mini_cluster.cc:1428] Setting key f1465a7368c5f5ad82e089b0afa9706e
I20260619 02:50:11.768368 30546 heartbeater.cc:511] Master 127.21.164.254:34133 requested a full tablet report, sending...
I20260619 02:50:11.768851 30355 ts_manager.cc:194] Registered new tserver with Master: c70fe76e31174e92b0fcce6b39fb8113 (127.21.164.193:37551)
I20260619 02:50:11.769443 30355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:48945
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:50:11.790998 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34133
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:11.871497 30551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:11.871711 30551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:11.871732 30551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:11.872982 30551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:11.873023 30551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:11.873052 30551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:50:11.874516 30551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:11.874737 30551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:11.874943 30551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:11.879719 30557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:11.879679 30556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:11.879671 30559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:11.879922 30551 server_base.cc:1034] running on GCE node
I20260619 02:50:11.880105 30551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:11.880301 30551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:11.881492 30551 hybrid_clock.cc:648] HybridClock initialized: now 1781837411881474 us; error 29 us; skew 500 ppm
Jun 19 02:50:11 dist-test-slave-jt0t krb5kdc[30310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:11.883894 30551 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:50:11.884331 30551 webserver.cc:460] Webserver started at http://127.21.164.194:43153/ using document root <none> and password file <none>
I20260619 02:50:11.884572 30551 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:11.884624 30551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:11.884778 30551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:11.885860 30551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "e52def068a53483bb93aefc7c90e379f"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "b9284f7047b5232f22d98e49a8573f0d"
server_key_iv: "33b42318ea9bfd77ff5bb33b054f44e0"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.886188 30551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "e52def068a53483bb93aefc7c90e379f"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "b9284f7047b5232f22d98e49a8573f0d"
server_key_iv: "33b42318ea9bfd77ff5bb33b054f44e0"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.887377 30551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:50:11.888024 30566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.888173 30551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:11.888254 30551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "e52def068a53483bb93aefc7c90e379f"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "b9284f7047b5232f22d98e49a8573f0d"
server_key_iv: "33b42318ea9bfd77ff5bb33b054f44e0"
server_key_version: "encryptionkey@0"
I20260619 02:50:11.888314 30551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:11.918742 30551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:11.920975 30551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:11.921118 30551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:11.921342 30551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:11.921656 30551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:11.921690 30551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.921720 30551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:11.921742 30551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:11.927143 30551 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:38339
I20260619 02:50:11.927183 30679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:38339 every 8 connection(s)
I20260619 02:50:11.927515 30551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:50:11 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:50:11.935899 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.928201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:34717 (local address 127.21.164.254:34133)
0619 02:50:11.928260 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:11.928261 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:11.928439 (+ 178us) server_negotiation.cc:407] Connection header received
0619 02:50:11.928921 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:11.928922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:11.928931 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:11.928987 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:11.929917 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.930412 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.931345 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.931528 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.931571 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.933593 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:11.933595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:11.933595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:11.933622 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:11.934857 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.935197 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.935197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.935198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.935242 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:11.935414 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:11.935414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:11.935414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:11.935523 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:11.935567 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:11.935774 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 02:50:11.935832 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:50:11.936064 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 30551
I20260619 02:50:11.936118 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:11.928170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:34717)
0619 02:50:11.928314 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:11.928355 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:11.928843 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:11.929045 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:11.929049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:11.929624 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:11.929823 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.929828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.930486 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.930487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.931268 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:11.931270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:11.931580 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:11.931581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:11.931677 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:11.932144 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:11.932152 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:11.933512 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:11.934929 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:11.934935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:11.934937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:11.935145 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:11.935285 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:11.935285 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:11.935286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:11.935374 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:11.935564 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:11.935567 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:11.935681 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 02:50:11.935758 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:50:11.935889 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260619 02:50:11.936169 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:50:11.936339 22163 external_mini_cluster.cc:1428] Setting key 9302655a6d9f090508f3a463827d1527
I20260619 02:50:11.936709 30680 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34133
I20260619 02:50:11.936801 30680 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:11.936975 30680 heartbeater.cc:511] Master 127.21.164.254:34133 requested a full tablet report, sending...
I20260619 02:50:11.937314 30355 ts_manager.cc:194] Registered new tserver with Master: e52def068a53483bb93aefc7c90e379f (127.21.164.194:38339)
I20260619 02:50:11.937685 30355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:34717
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:50:11.960099 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34133
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:12.039340 30685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:12.039542 30685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:12.039564 30685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:12.040719 30685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:12.040750 30685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:12.040773 30685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:50:12.042137 30685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:12.042323 30685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:12.042500 30685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:12.047256 30693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:12.047250 30691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:12.047245 30690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:12.047484 30685 server_base.cc:1034] running on GCE node
I20260619 02:50:12.047675 30685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:12.047879 30685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:12.049019 30685 hybrid_clock.cc:648] HybridClock initialized: now 1781837412049000 us; error 30 us; skew 500 ppm
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837412, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:12.051369 30685 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:50:12.051828 30685 webserver.cc:460] Webserver started at http://127.21.164.195:44205/ using document root <none> and password file <none>
I20260619 02:50:12.052035 30685 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:12.052083 30685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:12.052204 30685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:12.053301 30685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "12c1fe653a4547a89586cdf91567cb59"
format_stamp: "Formatted at 2026-06-19 02:50:12 on dist-test-slave-jt0t"
server_key: "c8f4989f81727aefe98cf6d698f43504"
server_key_iv: "9473557113cc6fc09f425a23658d5d6c"
server_key_version: "encryptionkey@0"
I20260619 02:50:12.053632 30685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "12c1fe653a4547a89586cdf91567cb59"
format_stamp: "Formatted at 2026-06-19 02:50:12 on dist-test-slave-jt0t"
server_key: "c8f4989f81727aefe98cf6d698f43504"
server_key_iv: "9473557113cc6fc09f425a23658d5d6c"
server_key_version: "encryptionkey@0"
I20260619 02:50:12.054924 30685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:50:12.055639 30700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.055780 30685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:12.055864 30685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "12c1fe653a4547a89586cdf91567cb59"
format_stamp: "Formatted at 2026-06-19 02:50:12 on dist-test-slave-jt0t"
server_key: "c8f4989f81727aefe98cf6d698f43504"
server_key_iv: "9473557113cc6fc09f425a23658d5d6c"
server_key_version: "encryptionkey@0"
I20260619 02:50:12.055927 30685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:12.067559 30685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:12.069407 30685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:12.069545 30685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:12.069761 30685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:12.070055 30685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:12.070087 30685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.070116 30685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:12.070133 30685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.075556 30685 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:42247
I20260619 02:50:12.075611 30813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:42247 every 8 connection(s)
I20260619 02:50:12.075917 30685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837412, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:50:12.084741 30394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.076770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:39919 (local address 127.21.164.254:34133)
0619 02:50:12.076825 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:12.076827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:12.076893 (+ 66us) server_negotiation.cc:407] Connection header received
0619 02:50:12.077397 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:12.077398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:12.077407 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:12.077452 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:12.078403 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.079162 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.080160 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.080347 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.080400 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.082551 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:12.082553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:12.082553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:12.082560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:12.083639 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:12.083974 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:12.083975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:12.083975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:12.084022 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:12.084197 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:12.084198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:12.084198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:12.084295 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:12.084339 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:12.084549 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 02:50:12.084617 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:50:12.084880 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 30685
I20260619 02:50:12.084937 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.076595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:39919)
0619 02:50:12.076771 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:12.076820 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:12.077292 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:12.077516 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:12.077519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:12.078069 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:12.078302 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.078309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.079266 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.079267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.080079 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.080081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.080440 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.080441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.080555 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.081111 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:12.081121 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:12.082461 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:12.083708 (+ 1247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:12.083714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:12.083716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:12.083922 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:12.084066 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:12.084066 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:12.084066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:12.084155 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:12.084335 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:12.084337 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:12.084453 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 02:50:12.084530 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:50:12.084731 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260619 02:50:12.084964 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:50:12.085101 22163 external_mini_cluster.cc:1428] Setting key e2deb2b5ab5850c5c3a6dcfcb2de1f2e
I20260619 02:50:12.085516 30814 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34133
I20260619 02:50:12.085621 30814 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:12.085804 30814 heartbeater.cc:511] Master 127.21.164.254:34133 requested a full tablet report, sending...
I20260619 02:50:12.086148 30355 ts_manager.cc:194] Registered new tserver with Master: 12c1fe653a4547a89586cdf91567cb59 (127.21.164.195:42247)
I20260619 02:50:12.086514 30355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:39919
I20260619 02:50:12.096840 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab.
Entry for principal oryx/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab.
WARNING: no policy specified for oryx/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab.
Entry for principal oryx/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab.
WARNING: no policy specified for oryx/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab.
Entry for principal oryx/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab.
I20260619 02:50:12.161580 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 30417
I20260619 02:50:12.165505 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 30551
I20260619 02:50:12.168954 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 30685
I20260619 02:50:12.172822 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 30323
I20260619 02:50:12.177105 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34133
--webserver_interface=127.21.164.254
--webserver_port=42513
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:34133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:12.256969 30823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:12.257186 30823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:12.257208 30823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:12.258345 30823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:12.258383 30823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:12.258396 30823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:12.258409 30823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:12.258432 30823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:12.259892 30823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:34133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=42513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:12.260103 30823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:12.260293 30823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:12.264964 30828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:12.265004 30829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:12.264981 30831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:12.265051 30823 server_base.cc:1034] running on GCE node
I20260619 02:50:12.265414 30823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:12.265690 30823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:12.266844 30823 hybrid_clock.cc:648] HybridClock initialized: now 1781837412266832 us; error 26 us; skew 500 ppm
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837412, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:12.269237 30823 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:50:12.269625 30823 webserver.cc:460] Webserver started at http://127.21.164.254:42513/ using document root <none> and password file <none>
I20260619 02:50:12.269807 30823 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:12.269865 30823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:12.271117 30823 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:12.271765 30838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.271945 30823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:50:12.272047 30823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "dfb7cdb4e12e8a09226e58715fc07313"
server_key_iv: "2a4f25a6834211524873ad3e8e3749ef"
server_key_version: "encryptionkey@0"
I20260619 02:50:12.272385 30823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:12.282549 30823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:12.284293 30823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:12.284421 30823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:12.288067 30823 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:34133
I20260619 02:50:12.288107 30890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:34133 every 8 connection(s)
I20260619 02:50:12.288450 30823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:50:12.289825 30891 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:50:12.290405 30891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922: Bootstrap starting.
I20260619 02:50:12.291461 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 30823
I20260619 02:50:12.291726 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:37551
--local_ip_for_outbound_sockets=127.21.164.193
--tserver_master_addrs=127.21.164.254:34133
--webserver_port=33789
--webserver_interface=127.21.164.193
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.164.193
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:50:12.298182 30891 log.cc:824] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:12.299952 30891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:50:12.300209 30891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922: Bootstrap complete.
I20260619 02:50:12.302261 30891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } }
I20260619 02:50:12.302484 30891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee924ec0d4b4ea5bba3a3238d7ee922, State: Initialized, Role: FOLLOWER
I20260619 02:50:12.302635 30891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } }
I20260619 02:50:12.302699 30891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:50:12.302743 30891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:50:12.302789 30891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:50:12.303776 30891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } }
I20260619 02:50:12.303869 30891 leader_election.cc:302] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ee924ec0d4b4ea5bba3a3238d7ee922; no voters:
I20260619 02:50:12.304044 30891 leader_election.cc:288] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:50:12.304102 30895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:50:12.304219 30895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [term 2 LEADER]: Becoming Leader. State: Replica: 1ee924ec0d4b4ea5bba3a3238d7ee922, State: Running, Role: LEADER
I20260619 02:50:12.304298 30895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } }
I20260619 02:50:12.304319 30891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:12.304561 30896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } } }
I20260619 02:50:12.304694 30896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:12.304989 30897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee924ec0d4b4ea5bba3a3238d7ee922. Latest consensus state: current_term: 2 leader_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee924ec0d4b4ea5bba3a3238d7ee922" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34133 } } }
I20260619 02:50:12.305111 30897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:12.305192 30899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:12.305699 30899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:50:12.305842 30899 catalog_manager.cc:1259] Loaded cluster ID: c8d806ec61724273bf0f65b34981c8a1
I20260619 02:50:12.306092 30899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:50:12.306903 30899 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:12.307070 30899 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ee924ec0d4b4ea5bba3a3238d7ee922: Loaded TSK: 0
I20260619 02:50:12.307256 30899 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:50:12.381819 30893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:12.382030 30893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:12.382055 30893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:12.383368 30893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:12.383421 30893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:12.383457 30893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:50:12.385532 30893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.193.keytab
--principal=oryx/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:37551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=33789
--tserver_master_addrs=127.21.164.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:12.385807 30893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:12.386029 30893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:12.391122 30917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:12.391100 30920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:12.391100 30918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:12.391279 30893 server_base.cc:1034] running on GCE node
I20260619 02:50:12.391628 30893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:12.391887 30893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:12.393045 30893 hybrid_clock.cc:648] HybridClock initialized: now 1781837412393030 us; error 34 us; skew 500 ppm
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837412, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:12.395561 30893 init.cc:377] Logged in from keytab as oryx/127.21.164.193@KRBTEST.COM (short username oryx)
I20260619 02:50:12.396004 30893 webserver.cc:460] Webserver started at http://127.21.164.193:33789/ using document root <none> and password file <none>
I20260619 02:50:12.396221 30893 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:12.396286 30893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:12.397825 30893 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:12.398466 30927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.398636 30893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:12.398720 30893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "c70fe76e31174e92b0fcce6b39fb8113"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "db6c705942efdf87a8caa39a85835a44"
server_key_iv: "3357bb21636b763bc31fdf1e6d3e488f"
server_key_version: "encryptionkey@0"
I20260619 02:50:12.399075 30893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:12.439013 30893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:12.441215 30893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:12.441363 30893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:12.441594 30893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:12.441886 30893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:12.441918 30893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.441941 30893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:12.441963 30893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.447288 30893 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:37551
I20260619 02:50:12.447369 31040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:37551 every 8 connection(s)
I20260619 02:50:12.447710 30893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:12.455806 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.448446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:51301)
0619 02:50:12.448621 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:12.448696 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:12.449193 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:12.449602 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:12.449606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:12.450185 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:12.450376 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.450381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.451476 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.451477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.452270 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.452272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.452882 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.452883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.453013 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.453498 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:12.453506 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:12.455648 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
W20260619 02:50:12.456032 31041 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:34133 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.164.254:34133: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
W20260619 02:50:12.456177 31044 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:12.448481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:51301 (local address 127.21.164.254:34133)
0619 02:50:12.448690 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:12.448701 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:12.448785 (+ 84us) server_negotiation.cc:407] Connection header received
0619 02:50:12.449394 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:12.449400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:12.449442 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:12.449554 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:12.450505 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.451380 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.452362 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.452794 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.452894 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.455781 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:51301: BlockingRecv error: recv got EOF from 127.21.164.193:51301 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260619 02:50:12.457168 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 30893
I20260619 02:50:12.457451 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:38339
--local_ip_for_outbound_sockets=127.21.164.194
--tserver_master_addrs=127.21.164.254:34133
--webserver_port=43153
--webserver_interface=127.21.164.194
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.164.194
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:12.460536 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.456266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:35541)
0619 02:50:12.456326 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:12.456337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:12.456374 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:12.456498 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:12.456499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:12.456605 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:12.456680 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.456682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.457281 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.457282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.458031 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.458034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.458384 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.458385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.458448 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.458951 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:12.458955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:12.460465 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 02:50:12.460673 31044 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:12.456318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:35541 (local address 127.21.164.254:34133)
0619 02:50:12.456352 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:12.456354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:12.456365 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:50:12.456432 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:12.456433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:12.456442 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:12.456488 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:12.456732 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.457200 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.458127 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.458320 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.458374 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.460592 (+ 2218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:35541: BlockingRecv error: recv got EOF from 127.21.164.193:35541 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:12.465161 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.460868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:56469)
0619 02:50:12.460919 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:12.460938 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:12.460983 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:12.461116 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:12.461117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:12.461254 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:12.461290 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.461292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.461950 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.461951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.462732 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.462735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.463036 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.463037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.463089 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.463433 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:12.463438 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:12.465105 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 02:50:12.465289 31044 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:12.460925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:56469 (local address 127.21.164.254:34133)
0619 02:50:12.460965 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:12.460967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:12.460975 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:12.461046 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:12.461046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:12.461056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:12.461099 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:12.461349 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.461871 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.462803 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.462976 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.463020 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.465213 (+ 2193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:56469: BlockingRecv error: recv got EOF from 127.21.164.193:56469 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 02:50:12.465276 31041 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:50:12.541774 31045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:12.541985 31045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:12.542007 31045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:12.543124 31045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:12.543159 31045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:12.543183 31045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:50:12.544574 31045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.194.keytab
--principal=oryx/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:38339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=43153
--tserver_master_addrs=127.21.164.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:12.544800 31045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:12.544989 31045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:50:12.549757 31045 server_base.cc:1034] running on GCE node
W20260619 02:50:12.549681 31053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:12.549685 31051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:12.549687 31050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:12.550105 31045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:12.550335 31045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:12.551477 31045 hybrid_clock.cc:648] HybridClock initialized: now 1781837412551450 us; error 41 us; skew 500 ppm
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837412, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:12.553987 31045 init.cc:377] Logged in from keytab as oryx/127.21.164.194@KRBTEST.COM (short username oryx)
I20260619 02:50:12.554409 31045 webserver.cc:460] Webserver started at http://127.21.164.194:43153/ using document root <none> and password file <none>
I20260619 02:50:12.554592 31045 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:12.554644 31045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:12.555935 31045 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:12.556464 31060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.556604 31045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.556715 31045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "e52def068a53483bb93aefc7c90e379f"
format_stamp: "Formatted at 2026-06-19 02:50:11 on dist-test-slave-jt0t"
server_key: "b9284f7047b5232f22d98e49a8573f0d"
server_key_iv: "33b42318ea9bfd77ff5bb33b054f44e0"
server_key_version: "encryptionkey@0"
I20260619 02:50:12.557067 31045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:12.579252 31045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:12.581287 31045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:12.581420 31045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:12.581648 31045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:12.581948 31045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:12.581980 31045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.582003 31045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:12.582027 31045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.587401 31045 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:38339
I20260619 02:50:12.587458 31173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:38339 every 8 connection(s)
I20260619 02:50:12.587781 31045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:50:12.592361 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 31045
I20260619 02:50:12.592687 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:42247
--local_ip_for_outbound_sockets=127.21.164.195
--tserver_master_addrs=127.21.164.254:34133
--webserver_port=44205
--webserver_interface=127.21.164.195
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.164.195
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:50:12.595379 31044 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:12.588492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:50697 (local address 127.21.164.254:34133)
0619 02:50:12.588560 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:12.588562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:12.588766 (+ 204us) server_negotiation.cc:407] Connection header received
0619 02:50:12.589232 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:12.589233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:12.589247 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:12.589303 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:12.590219 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.590712 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.591786 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.591999 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.592067 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.595321 (+ 3254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:50697: BlockingRecv error: recv got EOF from 127.21.164.194:50697 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:50:12.595453 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.588474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:50697)
0619 02:50:12.588624 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:12.588691 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:12.589148 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:12.589366 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:12.589369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:12.589920 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:12.590113 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.590117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.590808 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.590809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.591700 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.591703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.592098 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.592099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.592216 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.592808 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:12.592819 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:12.595228 (+ 2409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
W20260619 02:50:12.595634 31174 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:34133 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.164.254:34133: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:12.599715 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.595836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:45249)
0619 02:50:12.595866 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:12.595876 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:12.595922 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:12.596037 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:12.596037 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:12.596156 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:12.596197 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.596198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.596843 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.596844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.597335 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.597337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.597651 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.597652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.597709 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.598095 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:12.598100 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:12.599668 (+ 1568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14}
W20260619 02:50:12.599841 31044 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:12.595895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:45249 (local address 127.21.164.254:34133)
0619 02:50:12.595941 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:12.595943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:12.595951 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:12.595976 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:12.595977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:12.595985 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:12.596026 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:12.596256 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.596774 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.597399 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.597597 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.597651 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.599780 (+ 2129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:45249: BlockingRecv error: recv got EOF from 127.21.164.194:45249 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:12.603953 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.600101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:42455)
0619 02:50:12.600140 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:12.600151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:12.600185 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:12.600294 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:12.600295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:12.600412 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:12.600436 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.600438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.601074 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.601075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.601555 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.601556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.601937 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.601938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.601988 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.602352 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:12.602357 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:12.603891 (+ 1534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 02:50:12.604061 31044 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:12.600162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:42455 (local address 127.21.164.254:34133)
0619 02:50:12.600204 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:12.600206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:12.600215 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:12.600233 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:12.600234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:12.600243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:12.600286 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:12.600493 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.601004 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.601624 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.601873 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.601920 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.603971 (+ 2051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:42455: BlockingRecv error: recv got EOF from 127.21.164.194:42455 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 02:50:12.604171 31174 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:50:12.673301 31177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:12.673518 31177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:12.673543 31177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:12.674783 31177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:12.674834 31177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:12.674866 31177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:50:12.676519 31177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/krb5kdc/oryx_127.21.164.195.keytab
--principal=oryx/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:42247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=44205
--tserver_master_addrs=127.21.164.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:12.676793 31177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:12.677026 31177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:12.681912 31183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:12.681945 31182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:12.681967 31177 server_base.cc:1034] running on GCE node
W20260619 02:50:12.681945 31185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:12.682296 31177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:12.682551 31177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:12.683692 31177 hybrid_clock.cc:648] HybridClock initialized: now 1781837412683679 us; error 29 us; skew 500 ppm
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837412, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:12.686126 31177 init.cc:377] Logged in from keytab as oryx/127.21.164.195@KRBTEST.COM (short username oryx)
I20260619 02:50:12.686580 31177 webserver.cc:460] Webserver started at http://127.21.164.195:44205/ using document root <none> and password file <none>
I20260619 02:50:12.686800 31177 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:12.686856 31177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:12.688174 31177 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:12.688685 31192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.688824 31177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.688906 31177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "12c1fe653a4547a89586cdf91567cb59"
format_stamp: "Formatted at 2026-06-19 02:50:12 on dist-test-slave-jt0t"
server_key: "c8f4989f81727aefe98cf6d698f43504"
server_key_iv: "9473557113cc6fc09f425a23658d5d6c"
server_key_version: "encryptionkey@0"
I20260619 02:50:12.689244 31177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:12.702389 31177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:12.704391 31177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:12.704530 31177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:12.704793 31177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:12.705124 31177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:12.705166 31177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.705196 31177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:12.705220 31177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:12.710742 31177 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:42247
I20260619 02:50:12.710789 31305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:42247 every 8 connection(s)
I20260619 02:50:12.711160 31177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:12.718093 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 31177
W20260619 02:50:12.718418 31044 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:12.711936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:51977 (local address 127.21.164.254:34133)
0619 02:50:12.712021 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:12.712023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:12.712231 (+ 208us) server_negotiation.cc:407] Connection header received
0619 02:50:12.712733 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:12.712733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:12.712743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:12.712793 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:12.713701 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.714245 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.715290 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.715479 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.715535 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.718351 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:51977: BlockingRecv error: recv got EOF from 127.21.164.195:51977 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:50:12.718426 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.711922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:51977)
0619 02:50:12.712093 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:12.712144 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:12.712612 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:12.712852 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:12.712855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:12.713422 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:12.713625 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.713631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.714339 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.714340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.715200 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.715203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.715542 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.715543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.715645 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.716095 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:12.716103 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:12.718252 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
W20260619 02:50:12.718845 31306 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:34133 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.164.254:34133: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:12.723395 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.719059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:48721)
0619 02:50:12.719101 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:12.719109 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:12.719139 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:12.719330 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:12.719331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:12.719460 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:12.719510 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.719512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.720134 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.720135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.720623 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.720625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.721011 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.721012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.721075 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.721434 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:12.721438 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:12.723323 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 02:50:12.723479 31310 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:12.719114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:48721 (local address 127.21.164.254:34133)
0619 02:50:12.719241 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:12.719243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:12.719248 (+ 5us) server_negotiation.cc:407] Connection header received
0619 02:50:12.719263 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:12.719264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:12.719271 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:12.719321 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:12.719580 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.720067 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.720722 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.720946 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.721001 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.723412 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:48721: BlockingRecv error: recv got EOF from 127.21.164.195:48721 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260619 02:50:12.725481 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.718601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.21.164.254:34133)
0619 02:50:12.718642 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:12.718643 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:12.718715 (+ 72us) server_negotiation.cc:407] Connection header received
0619 02:50:12.718796 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:12.718797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:12.718806 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:12.718852 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:12.719475 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.719982 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.720784 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.720982 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.721038 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.722188 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:12.722192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:12.722192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:12.722203 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:12.724070 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:12.724404 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:12.724407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:12.724408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:12.724505 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:12.724730 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:12.724731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:12.724731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:12.724954 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:12.725086 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:12.725179 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:50:12.725365 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":22}
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:12 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:12.727898 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:12.723822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:39875)
0619 02:50:12.723866 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:12.723878 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:12.723920 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:12.724036 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:12.724037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:12.724164 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:12.724194 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.724195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.724967 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.724968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.725672 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:12.725674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.726037 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.726038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:12.726091 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.726465 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:12.726469 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:12.727834 (+ 1365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 02:50:12.727967 31310 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:12.723872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:39875 (local address 127.21.164.254:34133)
0619 02:50:12.723920 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:12.723922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:12.723930 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:50:12.723976 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:12.723976 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:12.723984 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:12.724021 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:12.724265 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.724898 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.725751 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:12.725974 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:12.726033 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:12.727921 (+ 1888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:39875: BlockingRecv error: recv got EOF from 127.21.164.195:39875 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 02:50:12.728080 31306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 02:50:13 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:13 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:13.469934 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:13.465612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:38045)
0619 02:50:13.465733 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:13.465744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:13.465791 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:13.465931 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:13.465932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:13.466088 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:13.466147 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:13.466148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.466740 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.466740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:13.467278 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:13.467280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.467593 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.467594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:13.467649 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:13.468060 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:13.468065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:13.469860 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
W20260619 02:50:13.470017 31312 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:13.465676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:38045 (local address 127.21.164.254:34133)
0619 02:50:13.465793 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:13.465796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:13.465806 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:13.465830 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:13.465830 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:13.465840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:13.465898 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:13.466201 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.466684 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.467335 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.467514 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.467559 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:13.469949 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:38045: BlockingRecv error: recv got EOF from 127.21.164.193:38045 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
Jun 19 02:50:13 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:13 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:13.609055 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:13.604563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:45781)
0619 02:50:13.604711 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:13.604723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:13.604789 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:13.604970 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:13.604971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:13.605178 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:13.605229 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:13.605231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.605881 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.605882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:13.606536 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:13.606539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.606864 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.606865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:13.606922 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:13.607332 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:13.607336 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:13.608966 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
W20260619 02:50:13.609117 31312 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:13.604643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:45781 (local address 127.21.164.254:34133)
0619 02:50:13.604718 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:13.604719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:13.604784 (+ 65us) server_negotiation.cc:407] Connection header received
0619 02:50:13.604895 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:13.604897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:13.604909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:13.604959 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:13.605295 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.605799 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.606610 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.606799 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.606853 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:13.609062 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:45781: BlockingRecv error: recv got EOF from 127.21.164.194:45781 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 19 02:50:13 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:13 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:13.732712 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:13.728473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:57973)
0619 02:50:13.728614 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:13.728625 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:13.728702 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:13.728810 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:13.728811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:13.728968 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:13.729017 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:13.729019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.729607 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.729607 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:13.730157 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:13.730158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.730499 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.730500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:13.730542 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:13.730955 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:13.730959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:13.732606 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
W20260619 02:50:13.732784 31312 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:13.728548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:57973 (local address 127.21.164.254:34133)
0619 02:50:13.728598 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:13.728600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:13.728699 (+ 99us) server_negotiation.cc:407] Connection header received
0619 02:50:13.728755 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:13.728756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:13.728768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:13.728804 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:13.729066 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.729543 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.730236 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:13.730429 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:13.730483 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:13.732729 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:57973: BlockingRecv error: recv got EOF from 127.21.164.195:57973 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 19 02:50:14 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:14 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:14.474972 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:14.470450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:46111)
0619 02:50:14.470564 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:14.470576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:14.470650 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:14.470790 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:14.470791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:14.470954 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:14.471001 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:14.471002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.471703 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.471704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:14.472251 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:14.472253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.472599 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.472600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:14.472640 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:14.473054 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:14.473058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:14.474912 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
W20260619 02:50:14.475095 31316 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:14.470529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:46111 (local address 127.21.164.254:34133)
0619 02:50:14.470668 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:14.470670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:14.470681 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:50:14.470705 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:14.470705 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:14.470716 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:14.470780 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:14.471084 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.471626 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.472324 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.472543 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.472597 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:14.475021 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:46111: BlockingRecv error: recv got EOF from 127.21.164.193:46111 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
Jun 19 02:50:14 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:14 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:14.613929 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:14.609554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:57509)
0619 02:50:14.609678 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:14.609688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:14.609761 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:14.609877 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:14.609878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:14.610031 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:14.610079 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:14.610080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.610751 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.610752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:14.611295 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:14.611308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.611680 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.611681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:14.611732 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:14.612190 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:14.612194 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:14.613869 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
W20260619 02:50:14.614034 31316 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:14.609602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:57509 (local address 127.21.164.254:34133)
0619 02:50:14.609689 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:14.609691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:14.609752 (+ 61us) server_negotiation.cc:407] Connection header received
0619 02:50:14.609807 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:14.609808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:14.609819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:14.609866 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:14.610168 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.610671 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.611396 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.611593 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.611649 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:14.613959 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:57509: BlockingRecv error: recv got EOF from 127.21.164.194:57509 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 02:50:14 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:14 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:14.737845 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:14.733261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:48075)
0619 02:50:14.733413 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:14.733424 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:14.733490 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:14.733647 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:14.733648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:14.733857 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:14.733905 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:14.733908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.734589 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.734590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:14.735128 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:14.735130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.735494 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.735495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:14.735534 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:14.735949 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:14.735953 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:14.737767 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
W20260619 02:50:14.737957 31316 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:14.733451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:48075 (local address 127.21.164.254:34133)
0619 02:50:14.733519 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:14.733521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:14.733531 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:14.733563 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:14.733564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:14.733578 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:14.733632 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:14.733969 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.734527 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.735229 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:14.735419 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:14.735483 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:14.737882 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:48075: BlockingRecv error: recv got EOF from 127.21.164.195:48075 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 02:50:15 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:15 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:15.479987 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:15.475383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:59525)
0619 02:50:15.475513 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:15.475523 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:15.475570 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:15.475718 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:15.475719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:15.475880 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:15.475931 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:15.475932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.476684 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.476685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:15.477228 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:15.477231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.477601 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.477602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:15.477651 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:15.478061 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:15.478066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:15.479903 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
W20260619 02:50:15.480095 31320 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:15.475427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:59525 (local address 127.21.164.254:34133)
0619 02:50:15.475549 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:15.475551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:15.475575 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:50:15.475630 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:15.475630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:15.475642 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:15.475699 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:15.476020 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.476570 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.477313 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.477536 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.477593 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:15.480025 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:59525: BlockingRecv error: recv got EOF from 127.21.164.193:59525 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
Jun 19 02:50:15 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:15 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:15.618507 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:15.614316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:51255)
0619 02:50:15.614443 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:15.614454 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:15.614508 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:15.614622 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:15.614622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:15.614776 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:15.614822 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:15.614824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.615414 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.615415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:15.615953 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:15.615955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.616248 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.616249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:15.616296 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:15.616745 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:15.616749 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:15.618446 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
W20260619 02:50:15.618615 31320 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:15.614381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:51255 (local address 127.21.164.254:34133)
0619 02:50:15.614429 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:15.614431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:15.614498 (+ 67us) server_negotiation.cc:407] Connection header received
0619 02:50:15.614551 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:15.614552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:15.614563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:15.614610 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:15.614873 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.615360 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.616013 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.616187 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.616230 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:15.618555 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:51255: BlockingRecv error: recv got EOF from 127.21.164.194:51255 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 02:50:15 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:15 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:15.742760 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:15.738415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:39085)
0619 02:50:15.738545 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:15.738555 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:15.738622 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:15.738735 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:15.738736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:15.738897 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:15.738945 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:15.738947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.739613 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.739613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:15.740143 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:15.740144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.740489 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.740490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:15.740527 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:15.740961 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:15.740966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:15.742679 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
W20260619 02:50:15.742830 31320 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:15.738511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:39085 (local address 127.21.164.254:34133)
0619 02:50:15.738551 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:15.738553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:15.738618 (+ 65us) server_negotiation.cc:407] Connection header received
0619 02:50:15.738665 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:15.738665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:15.738677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:15.738723 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:15.739015 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.739536 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.740217 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:15.740432 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:15.740485 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:15.742774 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:39085: BlockingRecv error: recv got EOF from 127.21.164.195:39085 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Jun 19 02:50:16 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:16 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:16.485234 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:16.480584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:52781)
0619 02:50:16.480718 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:16.480731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:16.480816 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:16.480956 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:16.480957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:16.481111 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:16.481162 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:16.481164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.481893 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.481894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:16.482476 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:16.482480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.482849 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.482850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:16.482915 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:16.483369 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:16.483374 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:16.485174 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
W20260619 02:50:16.485414 31324 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:16.480701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:52781 (local address 127.21.164.254:34133)
0619 02:50:16.480833 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:16.480835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:16.480844 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:16.480870 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:16.480871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:16.480883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:16.480944 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:16.481248 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.481780 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.482559 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.482754 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.482816 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:16.485296 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:52781: BlockingRecv error: recv got EOF from 127.21.164.193:52781 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
Jun 19 02:50:16 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:16 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:16.628351 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:16.619026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:59955)
0619 02:50:16.619181 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:16.619192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:16.619262 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:16.619387 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:16.619388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:16.619543 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:16.619592 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:16.619593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.620230 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.620231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:16.620816 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:16.620819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.621167 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.621167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:16.621203 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:16.626536 (+ 5333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:16.626542 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:16.628289 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
W20260619 02:50:16.628490 31324 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:16.619105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:59955 (local address 127.21.164.254:34133)
0619 02:50:16.619172 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:16.619174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:16.619245 (+ 71us) server_negotiation.cc:407] Connection header received
0619 02:50:16.619317 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:16.619318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:16.619328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:16.619366 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:16.619655 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.620144 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.620902 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.621108 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.621163 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:16.628424 (+ 7261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:59955: BlockingRecv error: recv got EOF from 127.21.164.194:59955 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 02:50:16 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:16 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:16.747781 31326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:16.743236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:46105)
0619 02:50:16.743375 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:16.743386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:16.743462 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:16.743609 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:16.743610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:16.743777 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:16.743838 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:16.743839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.744500 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.744501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:16.745076 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:16.745078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.745439 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.745440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:16.745497 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:16.745910 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:16.745914 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:16.747712 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
W20260619 02:50:16.747884 31324 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:16.743303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:46105 (local address 127.21.164.254:34133)
0619 02:50:16.743369 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:16.743372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:16.743450 (+ 78us) server_negotiation.cc:407] Connection header received
0619 02:50:16.743533 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:16.743534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:16.743548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:16.743597 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:16.743913 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.744425 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.745153 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:16.745373 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:16.745430 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:16.747821 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:46105: BlockingRecv error: recv got EOF from 127.21.164.195:46105 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 02:50:17 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:17 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:17.490310 31327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:17.485708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:59483)
0619 02:50:17.485827 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:17.485837 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:17.485883 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:17.486042 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:17.486043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:17.486200 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:17.486248 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:17.486250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.486927 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.486928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:17.487459 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:17.487461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.487827 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.487828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:17.487874 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:17.488318 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:17.488322 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:17.490248 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
W20260619 02:50:17.490428 31328 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:17.485807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:59483 (local address 127.21.164.254:34133)
0619 02:50:17.485924 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:17.485926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:17.485935 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:17.485959 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:17.485960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:17.485970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:17.486033 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:17.486339 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.486858 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.487549 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.487753 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.487816 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:17.490346 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:59483: BlockingRecv error: recv got EOF from 127.21.164.193:59483 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
Jun 19 02:50:17 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:17 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:17.633811 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:17.629029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:41885)
0619 02:50:17.629206 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:17.629221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:17.629295 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:17.629445 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:17.629446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:17.629622 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:17.629673 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:17.629675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.630388 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.630389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:17.630988 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:17.630990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.631366 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.631367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:17.631405 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:17.632014 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:17.632019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:17.633747 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
W20260619 02:50:17.633932 31328 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:17.629222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:41885 (local address 127.21.164.254:34133)
0619 02:50:17.629319 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:17.629320 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:17.629330 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:17.629358 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:17.629359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:17.629369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:17.629427 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:17.629733 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.630306 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.631077 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.631295 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.631362 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:17.633861 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:41885: BlockingRecv error: recv got EOF from 127.21.164.194:41885 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 02:50:17 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:17 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:17.752620 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:17.748253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:38115)
0619 02:50:17.748421 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:17.748432 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:17.748496 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:17.748610 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:17.748610 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:17.748801 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:17.748852 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:17.748854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.749442 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.749443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:17.749954 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:17.749955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.750316 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.750318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:17.750371 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:17.750798 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:17.750803 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:17.752557 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
W20260619 02:50:17.752732 31328 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:17.748335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:38115 (local address 127.21.164.254:34133)
0619 02:50:17.748396 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:17.748398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:17.748489 (+ 91us) server_negotiation.cc:407] Connection header received
0619 02:50:17.748540 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:17.748540 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:17.748552 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:17.748598 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:17.748906 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.749387 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.750035 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:17.750239 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:17.750300 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:17.752644 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:38115: BlockingRecv error: recv got EOF from 127.21.164.195:38115 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 02:50:18 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:18 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:18.495380 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:18.490831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:53397)
0619 02:50:18.490997 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:18.491012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:18.491075 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:18.491198 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:18.491199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:18.491360 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:18.491407 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:18.491409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.492084 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.492085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:18.492753 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:18.492757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.493101 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.493102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:18.493160 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:18.493603 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:18.493607 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:18.495320 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
W20260619 02:50:18.495529 31332 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:18.490916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:53397 (local address 127.21.164.254:34133)
0619 02:50:18.491067 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:18.491069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:18.491078 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:18.491121 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:18.491122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:18.491131 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:18.491188 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:18.491517 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.492011 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.492830 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.493030 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.493080 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:18.495456 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:53397: BlockingRecv error: recv got EOF from 127.21.164.193:53397 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
Jun 19 02:50:18 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:18 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:18.638650 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:18.634348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:45985)
0619 02:50:18.634492 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:18.634506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:18.634579 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:18.634709 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:18.634710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:18.634876 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:18.634921 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:18.634923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.635579 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.635580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:18.636096 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:18.636097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.636414 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.636415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:18.636469 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:18.636942 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:18.636946 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:18.638574 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
W20260619 02:50:18.638729 31332 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:18.634468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:45985 (local address 127.21.164.254:34133)
0619 02:50:18.634520 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:18.634522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:18.634576 (+ 54us) server_negotiation.cc:407] Connection header received
0619 02:50:18.634637 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:18.634638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:18.634650 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:18.634696 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:18.635015 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.635505 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.636170 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.636355 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.636409 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:18.638675 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:45985: BlockingRecv error: recv got EOF from 127.21.164.194:45985 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 19 02:50:18 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:18 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:18.757472 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:18.753113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:41715)
0619 02:50:18.753223 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:18.753234 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:18.753288 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:18.753408 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:18.753409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:18.753564 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:18.753612 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:18.753613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.754242 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.754243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:18.754779 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:18.754781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.755100 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.755102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:18.755155 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:18.755655 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:18.755661 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:18.757385 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
W20260619 02:50:18.757536 31332 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:18.753192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:41715 (local address 127.21.164.254:34133)
0619 02:50:18.753270 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:18.753272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:18.753282 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:18.753350 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:18.753350 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:18.753362 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:18.753401 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:18.753664 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.754162 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.754840 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:18.755022 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:18.755065 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:18.757483 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:41715: BlockingRecv error: recv got EOF from 127.21.164.195:41715 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:50:19 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:19 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:19.500447 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:19.495851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:47497)
0619 02:50:19.496045 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:19.496057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:19.496121 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:19.496249 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:19.496250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:19.496408 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:19.496461 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:19.496464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.497132 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.497133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:19.497679 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:19.497681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.498075 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.498077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:19.498133 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:19.498593 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:19.498597 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:19.500384 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
W20260619 02:50:19.500566 31336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:19.495928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:47497 (local address 127.21.164.254:34133)
0619 02:50:19.496058 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:19.496060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:19.496113 (+ 53us) server_negotiation.cc:407] Connection header received
0619 02:50:19.496164 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:19.496164 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:19.496175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:19.496233 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:19.496524 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.497068 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.497765 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.497977 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.498039 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:19.500494 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:47497: BlockingRecv error: recv got EOF from 127.21.164.193:47497 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
Jun 19 02:50:19 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:19 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:19.643638 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:19.639173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:36263)
0619 02:50:19.639315 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:19.639328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:19.639388 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:19.639560 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:19.639561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:19.639724 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:19.639771 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:19.639773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.640405 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.640407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:19.640973 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:19.640975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.641338 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.641339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:19.641394 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:19.641852 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:19.641856 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:19.643562 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":59,"threads_started":1}
W20260619 02:50:19.643780 31336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:19.639287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:36263 (local address 127.21.164.254:34133)
0619 02:50:19.639355 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:19.639357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:19.639370 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:50:19.639470 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:19.639471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:19.639483 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:19.639534 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:19.639842 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.640333 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.641048 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.641264 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.641326 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:19.643660 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:36263: BlockingRecv error: recv got EOF from 127.21.164.194:36263 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 02:50:19 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:19 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:19.762342 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:19.757951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:38753)
0619 02:50:19.758079 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:19.758091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:19.758157 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:19.758284 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:19.758284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:19.758438 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:19.758520 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:19.758522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.759194 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.759194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:19.759767 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:19.759770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.760081 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.760082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:19.760135 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:19.760567 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:19.760572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:19.762263 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
W20260619 02:50:19.762445 31336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:19.758030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:38753 (local address 127.21.164.254:34133)
0619 02:50:19.758106 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:19.758108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:19.758151 (+ 43us) server_negotiation.cc:407] Connection header received
0619 02:50:19.758208 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:19.758209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:19.758222 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:19.758272 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:19.758609 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.759130 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.759835 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:19.760026 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:19.760077 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:19.762388 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:38753: BlockingRecv error: recv got EOF from 127.21.164.195:38753 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 02:50:20 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:20 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:20.505678 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:20.500981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:33889)
0619 02:50:20.501144 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:20.501157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:20.501231 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:20.501402 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:20.501403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:20.501558 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:20.501606 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:20.501608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.502322 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.502323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:20.502950 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:20.502952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.503320 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.503320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:20.503361 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:20.503752 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:20.503756 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:20.505589 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
W20260619 02:50:20.505744 31340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:20.501130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:33889 (local address 127.21.164.254:34133)
0619 02:50:20.501236 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:20.501238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:20.501247 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:20.501285 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:20.501286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:20.501297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:20.501345 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:20.501697 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.502257 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.503009 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.503258 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.503315 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:20.505684 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:33889: BlockingRecv error: recv got EOF from 127.21.164.193:33889 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":60,"threads_started":1}
Jun 19 02:50:20 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:20 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:20.648746 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:20.644105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:47649)
0619 02:50:20.644223 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:20.644233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:20.644287 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:20.644391 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:20.644391 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:20.644575 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:20.644624 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:20.644626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.645452 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.645453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:20.645988 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:20.645990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.646346 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.646348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:20.646411 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:20.646832 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:20.646837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:20.648681 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
W20260619 02:50:20.648836 31340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:20.644142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:47649 (local address 127.21.164.254:34133)
0619 02:50:20.644191 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:20.644193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:20.644277 (+ 84us) server_negotiation.cc:407] Connection header received
0619 02:50:20.644337 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:20.644338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:20.644348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:20.644385 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:20.644702 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.645345 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.646066 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.646261 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.646314 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:20.648776 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:47649: BlockingRecv error: recv got EOF from 127.21.164.194:47649 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 19 02:50:20 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:20 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:20.767001 31342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:20.762745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:42885)
0619 02:50:20.762889 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:20.762899 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:20.762958 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:20.763071 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:20.763072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:20.763224 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:20.763272 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:20.763274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.763928 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.763929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:20.764456 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:20.764458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.764829 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.764830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:20.764873 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:20.765328 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:20.765332 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:20.766924 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
W20260619 02:50:20.767073 31340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:20.762786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:42885 (local address 127.21.164.254:34133)
0619 02:50:20.762843 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:20.762845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:20.762941 (+ 96us) server_negotiation.cc:407] Connection header received
0619 02:50:20.762997 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:20.762997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:20.763014 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:20.763062 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:20.763357 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.763855 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.764526 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:20.764772 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:20.764826 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:20.767005 (+ 2179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:42885: BlockingRecv error: recv got EOF from 127.21.164.195:42885 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 02:50:21 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:21 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:21.510681 31343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:21.506125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:55103)
0619 02:50:21.506294 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:21.506305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:21.506359 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:21.506540 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:21.506542 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:21.506724 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:21.506773 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:21.506775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.507446 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.507447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:21.507995 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:21.507997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.508371 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.508372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:21.508431 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:21.508881 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:21.508886 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:21.510620 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
W20260619 02:50:21.510813 31344 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:21.506209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:55103 (local address 127.21.164.254:34133)
0619 02:50:21.506324 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:21.506326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:21.506346 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:50:21.506413 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:21.506414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:21.506425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:21.506476 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:21.506852 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.507356 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.508078 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.508275 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.508331 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:21.510729 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:55103: BlockingRecv error: recv got EOF from 127.21.164.193:55103 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
Jun 19 02:50:21 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:21 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:21.657136 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:21.649251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:53761)
0619 02:50:21.649376 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:21.649388 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:21.649437 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:21.649565 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:21.649565 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:21.649727 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:21.649773 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:21.649775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.650428 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.650429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:21.650983 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:21.650985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.651318 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.651319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:21.651385 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:21.655205 (+ 3820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:21.655211 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:21.657058 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
W20260619 02:50:21.657231 31344 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:21.649364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:53761 (local address 127.21.164.254:34133)
0619 02:50:21.649419 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:21.649421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:21.649432 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:50:21.649487 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:21.649488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:21.649498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:21.649554 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:21.649832 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.650341 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.651051 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.651237 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.651286 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:21.657169 (+ 5883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:53761: BlockingRecv error: recv got EOF from 127.21.164.194:53761 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 02:50:21 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:21 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:21.771843 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:21.767484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:36665)
0619 02:50:21.767637 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:21.767649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:21.767724 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:21.767871 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:21.767872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:21.768038 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:21.768083 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:21.768085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.768721 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.768722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:21.769244 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:21.769246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.769561 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.769562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:21.769610 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:21.770049 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:21.770053 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:21.771767 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
W20260619 02:50:21.771914 31344 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:21.767575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:36665 (local address 127.21.164.254:34133)
0619 02:50:21.767649 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:21.767651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:21.767718 (+ 67us) server_negotiation.cc:407] Connection header received
0619 02:50:21.767786 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:21.767787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:21.767802 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:21.767848 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:21.768158 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.768642 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.769314 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:21.769503 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:21.769561 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:21.771860 (+ 2299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:36665: BlockingRecv error: recv got EOF from 127.21.164.195:36665 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 02:50:22 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:22 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:22.516010 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:22.511198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:51211)
0619 02:50:22.511329 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:22.511375 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:22.511464 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:22.511669 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:22.511670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:22.511873 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:22.511929 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:22.511932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.512713 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.512714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:22.513259 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:22.513261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.513613 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.513614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:22.513669 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:22.514123 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:22.514127 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:22.515941 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
W20260619 02:50:22.516109 31348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:22.511245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:51211 (local address 127.21.164.254:34133)
0619 02:50:22.511405 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:22.511407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:22.511422 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:50:22.511535 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:22.511536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:22.511550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:22.511610 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:22.512073 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.512608 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.513332 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.513552 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.513608 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:22.516039 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:51211: BlockingRecv error: recv got EOF from 127.21.164.193:51211 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
Jun 19 02:50:22 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:22 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:22.661868 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:22.657598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:60425)
0619 02:50:22.657725 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:22.657736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:22.657790 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:22.657906 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:22.657907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:22.658066 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:22.658115 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:22.658117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.658740 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.658740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:22.659320 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:22.659322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.659697 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.659698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:22.659733 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:22.660165 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:22.660169 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:22.661790 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
W20260619 02:50:22.661952 31348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:22.657648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:60425 (local address 127.21.164.254:34133)
0619 02:50:22.657703 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:22.657705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:22.657785 (+ 80us) server_negotiation.cc:407] Connection header received
0619 02:50:22.657834 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:22.657835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:22.657846 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:22.657899 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:22.658164 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.658666 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.659410 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.659635 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.659694 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:22.661899 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:60425: BlockingRecv error: recv got EOF from 127.21.164.194:60425 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 02:50:22 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:22 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:22.776988 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:22.772386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:37319)
0619 02:50:22.772538 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:22.772554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:22.772612 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:22.772792 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:22.772793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:22.772974 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:22.773022 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:22.773024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.773650 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.773650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:22.774185 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:22.774187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.774467 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.774467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:22.774506 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:22.774939 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:22.774943 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:22.776894 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
W20260619 02:50:22.777056 31348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:22.772556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:37319 (local address 127.21.164.254:34133)
0619 02:50:22.772614 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:22.772617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:22.772627 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:22.772677 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:22.772678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:22.772689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:22.772744 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:22.773088 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.773591 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.774241 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:22.774417 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:22.774459 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:22.777005 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:37319: BlockingRecv error: recv got EOF from 127.21.164.195:37319 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 02:50:23 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:23 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:23.520991 31351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:23.516514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:48109)
0619 02:50:23.516704 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:23.516717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:23.516785 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:23.516911 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:23.516912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:23.517079 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:23.517122 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:23.517124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.517811 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.517812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:23.518461 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:23.518464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.518789 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.518790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:23.518824 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:23.519227 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:23.519231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:23.520910 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":107,"threads_started":1}
W20260619 02:50:23.521098 31352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:23.516599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:48109 (local address 127.21.164.254:34133)
0619 02:50:23.516727 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:23.516728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:23.516771 (+ 43us) server_negotiation.cc:407] Connection header received
0619 02:50:23.516829 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:23.516829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:23.516840 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:23.516902 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:23.517186 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.517755 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.518537 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.518733 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.518787 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:23.521020 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:48109: BlockingRecv error: recv got EOF from 127.21.164.193:48109 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
Jun 19 02:50:23 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:23 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:23.666882 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:23.662328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:32803)
0619 02:50:23.662477 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:23.662490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:23.662567 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:23.662699 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:23.662700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:23.662904 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:23.662973 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:23.662975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.663643 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.663644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:23.664205 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:23.664207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.664540 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.664542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:23.664604 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:23.665125 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:23.665131 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:23.666783 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
W20260619 02:50:23.666958 31352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:23.662407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:32803 (local address 127.21.164.254:34133)
0619 02:50:23.662500 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:23.662502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:23.662547 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:50:23.662630 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:23.662631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:23.662644 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:23.662692 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:23.663048 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.663564 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.664282 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.664480 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.664535 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:23.666904 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:32803: BlockingRecv error: recv got EOF from 127.21.164.194:32803 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 02:50:23 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:23 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:23.781672 31354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:23.777410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:48525)
0619 02:50:23.777584 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:23.777593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:23.777651 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:23.777761 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:23.777762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:23.777921 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:23.777966 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:23.777968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.778595 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.778595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:23.779134 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:23.779135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.779518 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.779519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:23.779553 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:23.779953 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:23.779957 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:23.781603 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":113,"threads_started":1}
W20260619 02:50:23.781761 31352 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:23.777458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:48525 (local address 127.21.164.254:34133)
0619 02:50:23.777517 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:23.777518 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:23.777645 (+ 127us) server_negotiation.cc:407] Connection header received
0619 02:50:23.777690 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:23.777690 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:23.777703 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:23.777747 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:23.778028 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.778534 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.779224 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:23.779451 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:23.779511 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:23.781697 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:48525: BlockingRecv error: recv got EOF from 127.21.164.195:48525 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 02:50:24 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:24 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:24.526041 31355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:24.521454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:57405)
0619 02:50:24.521573 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:24.521584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:24.521648 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:24.521847 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:24.521848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:24.522023 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:24.522068 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:24.522070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.522705 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.522706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:24.523241 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:24.523243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.523581 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.523582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:24.523619 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:24.524101 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:24.524105 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:24.525979 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
W20260619 02:50:24.526140 31356 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:24.521568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:57405 (local address 127.21.164.254:34133)
0619 02:50:24.521672 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:24.521674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:24.521684 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:50:24.521709 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:24.521710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:24.521720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:24.521780 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:24.522155 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.522649 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.523316 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.523525 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.523577 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:24.526062 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:57405: BlockingRecv error: recv got EOF from 127.21.164.193:57405 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":45,"threads_started":1}
Jun 19 02:50:24 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:24 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:24.671845 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:24.667389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:48605)
0619 02:50:24.667532 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:24.667544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:24.667600 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:24.667749 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:24.667750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:24.667913 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:24.667957 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:24.667958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.668619 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.668619 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:24.669187 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:24.669189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.669520 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.669522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:24.669580 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:24.670030 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:24.670036 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:24.671784 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
W20260619 02:50:24.671949 31356 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:24.667476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:48605 (local address 127.21.164.254:34133)
0619 02:50:24.667547 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:24.667549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:24.667592 (+ 43us) server_negotiation.cc:407] Connection header received
0619 02:50:24.667661 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:24.667662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:24.667677 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:24.667741 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:24.668020 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.668559 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.669246 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.669433 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.669478 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:24.671888 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:48605: BlockingRecv error: recv got EOF from 127.21.164.194:48605 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 02:50:24 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:24 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:24.786841 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:24.782239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:42145)
0619 02:50:24.782401 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:24.782411 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:24.782465 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:24.782610 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:24.782611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:24.782771 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:24.782815 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:24.782817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.783509 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.783510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:24.784085 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:24.784087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.784451 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.784452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:24.784495 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:24.784946 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:24.784952 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:24.786778 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
W20260619 02:50:24.786928 31356 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:24.782340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:42145 (local address 127.21.164.254:34133)
0619 02:50:24.782401 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:24.782429 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:24.782452 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:50:24.782512 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:24.782512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:24.782527 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:24.782599 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:24.782910 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.783410 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.784184 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:24.784378 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:24.784445 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:24.786869 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:42145: BlockingRecv error: recv got EOF from 127.21.164.195:42145 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 02:50:25 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:25 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:25.531397 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:25.526632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:40473)
0619 02:50:25.526767 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:25.526778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:25.526841 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:25.526963 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:25.526964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:25.527168 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:25.527226 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:25.527229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.528108 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.528109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:25.528715 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:25.528718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.529102 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.529103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:25.529156 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:25.529648 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:25.529652 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:25.531307 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
W20260619 02:50:25.531494 31360 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:25.526729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:40473 (local address 127.21.164.254:34133)
0619 02:50:25.526830 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:25.526832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:25.526841 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:25.526890 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:25.526891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:25.526901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:25.526958 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:25.527305 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.528018 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.528799 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.529006 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.529065 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:25.531421 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:40473: BlockingRecv error: recv got EOF from 127.21.164.193:40473 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":49,"threads_started":1}
Jun 19 02:50:25 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:25 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:25.676795 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:25.672352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:36947)
0619 02:50:25.672516 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:25.672529 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:25.672590 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:25.672749 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:25.672749 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:25.672921 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:25.672963 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:25.672965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.673603 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.673603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:25.674124 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:25.674126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.674481 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.674483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:25.674532 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:25.674996 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:25.675001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:25.676733 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":58,"threads_started":1}
W20260619 02:50:25.676919 31360 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:25.672507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:36947 (local address 127.21.164.254:34133)
0619 02:50:25.672575 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:25.672577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:25.672586 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:25.672672 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:25.672673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:25.672685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:25.672739 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:25.673040 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.673533 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.674201 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.674398 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.674452 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:25.676847 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:36947: BlockingRecv error: recv got EOF from 127.21.164.194:36947 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 02:50:25 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:25 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:25.791864 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:25.787367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:60303)
0619 02:50:25.787540 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:25.787551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:25.787619 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:25.787766 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:25.787767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:25.787956 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:25.788003 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:25.788006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.788634 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.788634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:25.789206 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:25.789208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.789537 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.789538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:25.789591 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:25.790042 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:25.790047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:25.791773 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
W20260619 02:50:25.791919 31360 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:25.787455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:60303 (local address 127.21.164.254:34133)
0619 02:50:25.787513 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:25.787515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:25.787609 (+ 94us) server_negotiation.cc:407] Connection header received
0619 02:50:25.787687 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:25.787688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:25.787703 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:25.787750 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:25.788076 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.788562 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.789280 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:25.789477 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:25.789532 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:25.791864 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:60303: BlockingRecv error: recv got EOF from 127.21.164.195:60303 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 02:50:26 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:26 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:26.536227 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:26.531845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:58491)
0619 02:50:26.531990 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:26.532002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:26.532057 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:26.532179 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:26.532180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:26.532339 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:26.532416 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:26.532417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.533136 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.533137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:26.533687 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:26.533689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.533994 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.533994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:26.534032 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:26.534514 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:26.534518 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:26.536165 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
W20260619 02:50:26.536334 31364 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:26.531936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:58491 (local address 127.21.164.254:34133)
0619 02:50:26.532048 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:26.532049 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:26.532060 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:50:26.532103 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:26.532104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:26.532114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:26.532169 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:26.532501 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.533048 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.533752 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.533941 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.533987 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:26.536260 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:58491: BlockingRecv error: recv got EOF from 127.21.164.193:58491 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
Jun 19 02:50:26 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:26 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:26.684435 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:26.677273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:32831)
0619 02:50:26.677406 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:26.677417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:26.677474 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:26.677600 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:26.677600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:26.677754 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:26.677799 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:26.677801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.678401 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.678402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:26.678946 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:26.678949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.679276 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.679276 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:26.679314 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:26.682631 (+ 3317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:26.682637 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:26.684364 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
W20260619 02:50:26.684545 31364 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:26.677364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:32831 (local address 127.21.164.254:34133)
0619 02:50:26.677443 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:26.677445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:26.677459 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:50:26.677531 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:26.677531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:26.677541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:26.677589 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:26.677854 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.678345 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.679021 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.679218 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.679271 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:26.684479 (+ 5208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:32831: BlockingRecv error: recv got EOF from 127.21.164.194:32831 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 02:50:26 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:26 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.195@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:26.796808 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:26.792350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.195:46919)
0619 02:50:26.792494 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:26.792504 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:26.792560 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:26.792745 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:26.792746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:26.792924 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:26.792969 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:26.792970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.793604 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.793606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:26.794126 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:26.794128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.794447 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.794448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:26.794501 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:26.794920 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:26.794923 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:26.796745 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
W20260619 02:50:26.796890 31364 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:26.792479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:46919 (local address 127.21.164.254:34133)
0619 02:50:26.792534 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:26.792536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:26.792548 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:50:26.792629 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:26.792630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:26.792669 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:26.792714 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:26.793040 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.793536 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.794201 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:26.794383 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:26.794444 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:26.796831 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.195:46919: BlockingRecv error: recv got EOF from 127.21.164.195:46919 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 02:50:27 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:27 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.193@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:27.541399 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:27.536838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.193:52715)
0619 02:50:27.536988 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:27.537000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:27.537072 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:27.537189 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:27.537189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:27.537343 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:27.537395 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:27.537397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:27.538078 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:27.538079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:27.538636 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:27.538639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:27.539006 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:27.539006 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:27.539055 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:27.539554 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:27.539559 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:27.541327 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
W20260619 02:50:27.541543 31368 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:27.536878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:52715 (local address 127.21.164.254:34133)
0619 02:50:27.537001 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:27.537003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:27.537056 (+ 53us) server_negotiation.cc:407] Connection header received
0619 02:50:27.537118 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:27.537118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:27.537128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:27.537183 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:27.537460 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:27.537975 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:27.538739 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:27.538945 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:27.539001 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:27.541459 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.193:52715: BlockingRecv error: recv got EOF from 127.21.164.193:52715 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
Jun 19 02:50:27 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:50:27 dist-test-slave-jt0t krb5kdc[30310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.164.194@KRBTEST.COM for oryx/127.21.164.254@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:50:27.689419 31369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:27.684990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34133 (local address 127.21.164.194:39477)
0619 02:50:27.685114 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:27.685127 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:27.685203 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:27.685329 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:27.685330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:27.685509 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:27.685555 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:27.685557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:27.686197 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:27.686199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:27.686718 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:27.686719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:27.687058 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:27.687059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:27.687112 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:27.687566 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:27.687571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:27.689341 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.164.254:34133: Server oryx/127.21.164.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
W20260619 02:50:27.689503 31368 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:27.685069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:39477 (local address 127.21.164.254:34133)
0619 02:50:27.685143 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:27.685146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:27.685185 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:50:27.685259 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:27.685260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:27.685271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:27.685319 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:27.685636 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:27.686127 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:27.686795 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:27.686999 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:27.687053 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:27.689450 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.164.194:39477: BlockingRecv error: recv got EOF from 127.21.164.194:39477 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:50:27.723184 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 30893
I20260619 02:50:27.727268 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 31045
I20260619 02:50:27.731107 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 31177
I20260619 02:50:27.734778 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 30823
2026-06-19T02:50:27Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17360 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T02:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:27Z Disabled control of system clock
I20260619 02:50:27.760013 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34701
--webserver_interface=127.21.164.254
--webserver_port=0
--builtin_ntp_servers=127.21.164.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:34701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:50:27.838958 31373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:27.839201 31373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:27.839221 31373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:27.840399 31373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:27.840435 31373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:27.840446 31373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:27.840457 31373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:27.840468 31373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:27.841789 31373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:34701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:27.841995 31373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:27.842244 31373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:27.847098 31379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:27.847148 31373 server_base.cc:1034] running on GCE node
W20260619 02:50:27.847162 31381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:27.847162 31378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:27.847587 31373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:27.847889 31373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:27.849041 31373 hybrid_clock.cc:648] HybridClock initialized: now 1781837427849025 us; error 29 us; skew 500 ppm
I20260619 02:50:27.850198 31373 webserver.cc:460] Webserver started at http://127.21.164.254:32835/ using document root <none> and password file <none>
I20260619 02:50:27.850402 31373 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:27.850449 31373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:27.850576 31373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:27.851629 31373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "c8afaf2d32ad487da7516b89e6a238e4"
format_stamp: "Formatted at 2026-06-19 02:50:27 on dist-test-slave-jt0t"
server_key: "22ace5726e03e3a2e291d248c1a8e69a"
server_key_iv: "171b33daaddfb5610ca1d51b5885edd8"
server_key_version: "encryptionkey@0"
I20260619 02:50:27.851955 31373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "c8afaf2d32ad487da7516b89e6a238e4"
format_stamp: "Formatted at 2026-06-19 02:50:27 on dist-test-slave-jt0t"
server_key: "22ace5726e03e3a2e291d248c1a8e69a"
server_key_iv: "171b33daaddfb5610ca1d51b5885edd8"
server_key_version: "encryptionkey@0"
I20260619 02:50:27.853381 31373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:50:27.854103 31387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:27.854247 31373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:50:27.854331 31373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "c8afaf2d32ad487da7516b89e6a238e4"
format_stamp: "Formatted at 2026-06-19 02:50:27 on dist-test-slave-jt0t"
server_key: "22ace5726e03e3a2e291d248c1a8e69a"
server_key_iv: "171b33daaddfb5610ca1d51b5885edd8"
server_key_version: "encryptionkey@0"
I20260619 02:50:27.854394 31373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:27.872167 31373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:27.872503 31373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:27.872642 31373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:27.876296 31373 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:34701
I20260619 02:50:27.876331 31439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:34701 every 8 connection(s)
I20260619 02:50:27.876698 31373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:50:27.877344 31440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:27.879901 31440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4: Bootstrap starting.
I20260619 02:50:27.880455 31440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:27.880715 31440 log.cc:824] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:27.881351 31440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4: No bootstrap required, opened a new log
I20260619 02:50:27.883139 31440 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8afaf2d32ad487da7516b89e6a238e4" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34701 } }
I20260619 02:50:27.883333 31440 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:27.883368 31440 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8afaf2d32ad487da7516b89e6a238e4, State: Initialized, Role: FOLLOWER
I20260619 02:50:27.883522 31440 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8afaf2d32ad487da7516b89e6a238e4" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34701 } }
I20260619 02:50:27.883605 31440 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:50:27.883631 31440 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:50:27.883666 31440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:27.884106 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 31373
I20260619 02:50:27.884198 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:50:27.884256 31440 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8afaf2d32ad487da7516b89e6a238e4" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34701 } }
I20260619 02:50:27.884361 31440 leader_election.cc:302] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8afaf2d32ad487da7516b89e6a238e4; no voters:
I20260619 02:50:27.884392 22163 external_mini_cluster.cc:1428] Setting key 0886cf584429c988c8bbf862eb82ccb0
I20260619 02:50:27.884532 31440 leader_election.cc:288] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:50:27.884574 31443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:27.884815 31443 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [term 1 LEADER]: Becoming Leader. State: Replica: c8afaf2d32ad487da7516b89e6a238e4, State: Running, Role: LEADER
I20260619 02:50:27.885079 31443 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8afaf2d32ad487da7516b89e6a238e4" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34701 } }
I20260619 02:50:27.884860 31440 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:27.885599 31445 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8afaf2d32ad487da7516b89e6a238e4. Latest consensus state: current_term: 1 leader_uuid: "c8afaf2d32ad487da7516b89e6a238e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8afaf2d32ad487da7516b89e6a238e4" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34701 } } }
I20260619 02:50:27.885675 31445 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:27.886117 31444 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8afaf2d32ad487da7516b89e6a238e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8afaf2d32ad487da7516b89e6a238e4" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34701 } } }
I20260619 02:50:27.886162 31444 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4 [sys.catalog]: This master's current role is: LEADER
W20260619 02:50:27.887002 31461 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:50:27.887090 31461 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:50:27.887169 31451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:27.887311 31451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:50:27.888773 31451 catalog_manager.cc:1347] Generated new cluster ID: 714dfb86d2254212aea388c808b5f608
I20260619 02:50:27.888825 31451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:50:27.889763 31447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:27.884991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.21.164.254:34701)
0619 02:50:27.885193 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:27.885203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:27.885235 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:50:27.885357 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:27.885365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:27.885443 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:27.885572 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:27.885895 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:27.887095 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:27.887902 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:27.888324 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:27.888430 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:27.889151 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:27.889155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:27.889155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:27.889312 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:27.889345 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:27.889355 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:27.889408 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:27.889482 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 02:50:27.889560 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260619 02:50:27.901634 31451 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:50:27.902154 31451 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:27.909416 31451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8afaf2d32ad487da7516b89e6a238e4: Generated new TSK 0
I20260619 02:50:27.909668 31451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:50:27.915843 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34701
--builtin_ntp_servers=127.21.164.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 02:50:27.996420 31464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:27.996667 31464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:27.996701 31464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:27.997865 31464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:27.997910 31464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:27.997946 31464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:50:27.999306 31464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:27.999553 31464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:27.999840 31464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:28.004633 31470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.004606 31469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.004613 31472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:28.005117 31464 server_base.cc:1034] running on GCE node
I20260619 02:50:28.005319 31464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:28.005541 31464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:28.006678 31464 hybrid_clock.cc:648] HybridClock initialized: now 1781837428006666 us; error 29 us; skew 500 ppm
I20260619 02:50:28.007872 31464 webserver.cc:460] Webserver started at http://127.21.164.193:43091/ using document root <none> and password file <none>
I20260619 02:50:28.008097 31464 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:28.008145 31464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:28.008266 31464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:28.009354 31464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "f8cedd9927f24241835d1ede425de6fa"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "1acc7e66f4eaa0d8ab256e7152aecb05"
server_key_iv: "985750b5af45ae776a8662866eb8dd35"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.009680 31464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "f8cedd9927f24241835d1ede425de6fa"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "1acc7e66f4eaa0d8ab256e7152aecb05"
server_key_iv: "985750b5af45ae776a8662866eb8dd35"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.010857 31464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:50:28.011561 31478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.011719 31464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:28.011801 31464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "f8cedd9927f24241835d1ede425de6fa"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "1acc7e66f4eaa0d8ab256e7152aecb05"
server_key_iv: "985750b5af45ae776a8662866eb8dd35"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.011866 31464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:28.021613 31464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:28.021904 31464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:28.022040 31464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:28.022244 31464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:28.022552 31464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:28.022583 31464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.022610 31464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:28.022635 31464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.028112 31464 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:35025
I20260619 02:50:28.028162 31591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:35025 every 8 connection(s)
I20260619 02:50:28.028477 31464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:50:28.030581 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 31464
I20260619 02:50:28.030659 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:50:28.030826 22163 external_mini_cluster.cc:1428] Setting key 30e6544cdec08af2810f445b7884e12f
I20260619 02:50:28.031741 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34701
--builtin_ntp_servers=127.21.164.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 02:50:28.034163 31447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.029196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:55151 (local address 127.21.164.254:34701)
0619 02:50:28.029255 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:28.029257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:28.029486 (+ 229us) server_negotiation.cc:407] Connection header received
0619 02:50:28.030051 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:28.030052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:28.030067 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:28.030121 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:28.030792 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.031400 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.032807 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.033032 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.033117 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:28.033921 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:28.033922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:28.033923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:28.033928 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:28.033943 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:28.033948 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:28.033998 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:28.034076 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 02:50:28.034103 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 02:50:28.034415 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.029172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34701 (local address 127.21.164.193:55151)
0619 02:50:28.029371 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:28.029405 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:28.029933 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:28.030196 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:28.030200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:28.030209 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:28.030714 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:28.030720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.031495 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.031496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:28.032723 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:28.032727 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.033135 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.033136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:28.033250 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:28.033778 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:28.033779 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:28.033809 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:28.033839 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:28.034006 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:28.034010 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:28.034014 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:28.034077 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 02:50:28.034183 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260619 02:50:28.034912 31592 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34701
I20260619 02:50:28.035013 31592 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:28.035208 31592 heartbeater.cc:511] Master 127.21.164.254:34701 requested a full tablet report, sending...
I20260619 02:50:28.035724 31404 ts_manager.cc:194] Registered new tserver with Master: f8cedd9927f24241835d1ede425de6fa (127.21.164.193:35025)
I20260619 02:50:28.036315 31404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.193:55151
W20260619 02:50:28.114964 31595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:28.115239 31595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:28.115271 31595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:28.116803 31595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:28.116853 31595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:28.116881 31595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:50:28.118311 31595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:28.118566 31595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:28.118862 31595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:50:28.123688 31595 server_base.cc:1034] running on GCE node
W20260619 02:50:28.123672 31603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.123667 31601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.123659 31600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:28.124097 31595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:28.124310 31595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:28.125458 31595 hybrid_clock.cc:648] HybridClock initialized: now 1781837428125446 us; error 26 us; skew 500 ppm
I20260619 02:50:28.126796 31595 webserver.cc:460] Webserver started at http://127.21.164.194:46095/ using document root <none> and password file <none>
I20260619 02:50:28.127053 31595 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:28.127099 31595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:28.127209 31595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:28.128278 31595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "1acb2e4871b747b29c15b2f542607755"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "aace14fe3f513379dc7cfff4d673f47a"
server_key_iv: "2d1f7c571e4dfdd9c14818115785d590"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.128600 31595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "1acb2e4871b747b29c15b2f542607755"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "aace14fe3f513379dc7cfff4d673f47a"
server_key_iv: "2d1f7c571e4dfdd9c14818115785d590"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.129832 31595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:28.130527 31609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.130698 31595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:28.130781 31595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "1acb2e4871b747b29c15b2f542607755"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "aace14fe3f513379dc7cfff4d673f47a"
server_key_iv: "2d1f7c571e4dfdd9c14818115785d590"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.130836 31595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:28.139782 31595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:28.140064 31595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:28.140162 31595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:28.140343 31595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:28.140697 31595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:28.140735 31595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.140766 31595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:28.140789 31595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.146003 31595 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:33897
I20260619 02:50:28.146091 31722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:33897 every 8 connection(s)
I20260619 02:50:28.146377 31595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:50:28.146996 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 31595
I20260619 02:50:28.147099 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:50:28.147306 22163 external_mini_cluster.cc:1428] Setting key 80e43ed4157b1953f656d5defc59de50
I20260619 02:50:28.148262 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34701
--builtin_ntp_servers=127.21.164.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 02:50:28.151856 31447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.147076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:52955 (local address 127.21.164.254:34701)
0619 02:50:28.147133 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:28.147135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:28.147416 (+ 281us) server_negotiation.cc:407] Connection header received
0619 02:50:28.147973 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:28.147974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:28.147985 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:28.148031 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:28.148925 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.149402 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.150396 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.150611 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.150666 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:28.151605 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:28.151607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:28.151607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:28.151611 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:28.151625 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:28.151630 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:28.151671 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:28.151745 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 02:50:28.151775 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:50:28.152065 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.147069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34701 (local address 127.21.164.194:52955)
0619 02:50:28.147266 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:28.147314 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:28.147878 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:28.148113 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:28.148118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:28.148129 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:28.148837 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:28.148845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.149494 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.149496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:28.150312 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:28.150315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.150679 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.150679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:28.150809 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:28.151446 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:28.151446 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:28.151476 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:28.151501 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:28.151678 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:28.151683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:28.151686 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:28.151746 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:50:28.151856 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260619 02:50:28.152500 31723 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34701
I20260619 02:50:28.152614 31723 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:28.152839 31723 heartbeater.cc:511] Master 127.21.164.254:34701 requested a full tablet report, sending...
I20260619 02:50:28.153247 31404 ts_manager.cc:194] Registered new tserver with Master: 1acb2e4871b747b29c15b2f542607755 (127.21.164.194:33897)
I20260619 02:50:28.153734 31404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.194:52955
W20260619 02:50:28.229902 31726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:28.230125 31726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:28.230145 31726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:28.231264 31726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:28.231297 31726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:28.231321 31726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:50:28.232626 31726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:28.232851 31726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:28.233105 31726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:28.237919 31732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.237912 31731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.237912 31734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:28.238269 31726 server_base.cc:1034] running on GCE node
I20260619 02:50:28.238445 31726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:28.238653 31726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:28.239785 31726 hybrid_clock.cc:648] HybridClock initialized: now 1781837428239773 us; error 29 us; skew 500 ppm
I20260619 02:50:28.241029 31726 webserver.cc:460] Webserver started at http://127.21.164.195:36963/ using document root <none> and password file <none>
I20260619 02:50:28.241245 31726 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:28.241289 31726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:28.241415 31726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:28.242470 31726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "6e527046c2554fd984ec4783ddfb9ce0"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "83c1790f84fa78c3342d22cb54fd64cb"
server_key_iv: "6f069ba3fd2900f241bb115c850a1f89"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.242820 31726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "6e527046c2554fd984ec4783ddfb9ce0"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "83c1790f84fa78c3342d22cb54fd64cb"
server_key_iv: "6f069ba3fd2900f241bb115c850a1f89"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.244026 31726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:50:28.244752 31740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.244902 31726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:28.244979 31726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "6e527046c2554fd984ec4783ddfb9ce0"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "83c1790f84fa78c3342d22cb54fd64cb"
server_key_iv: "6f069ba3fd2900f241bb115c850a1f89"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.245054 31726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:28.270915 31726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:28.271224 31726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:28.271348 31726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:28.271576 31726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:28.271904 31726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:28.271937 31726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.271967 31726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:28.271991 31726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.277276 31726 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:33983
I20260619 02:50:28.277340 31853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:33983 every 8 connection(s)
I20260619 02:50:28.277640 31726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:50:28.282433 31447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.278379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:57557 (local address 127.21.164.254:34701)
0619 02:50:28.278443 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:28.278444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:28.278622 (+ 178us) server_negotiation.cc:407] Connection header received
0619 02:50:28.279153 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:28.279154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:28.279163 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:28.279220 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:28.279838 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.280343 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.281278 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.281462 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.281515 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:28.282220 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:28.282221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:28.282222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:28.282226 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:28.282240 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:28.282244 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:28.282283 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:28.282344 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 02:50:28.282370 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:50:28.282652 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.278332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34701 (local address 127.21.164.195:57557)
0619 02:50:28.278500 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:28.278541 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:28.279073 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:28.279274 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:28.279277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:28.279286 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:28.279743 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:28.279747 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.280409 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.280409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:28.281212 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:28.281213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:28.281520 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:28.281520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:28.281622 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:28.282096 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:28.282097 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:28.282125 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:28.282148 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:28.282290 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:28.282292 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:28.282295 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:28.282341 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:50:28.282439 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260619 02:50:28.283097 31854 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34701
I20260619 02:50:28.283195 31854 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:28.283249 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 31726
I20260619 02:50:28.283342 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:50:28.283396 31854 heartbeater.cc:511] Master 127.21.164.254:34701 requested a full tablet report, sending...
I20260619 02:50:28.283553 22163 external_mini_cluster.cc:1428] Setting key a9eb5325aed052e91e0708e17ed74ee1
I20260619 02:50:28.283756 31404 ts_manager.cc:194] Registered new tserver with Master: 6e527046c2554fd984ec4783ddfb9ce0 (127.21.164.195:33983)
I20260619 02:50:28.284184 31404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.195:57557
I20260619 02:50:28.284332 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:50:28.286266 31447 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:28.285816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.21.164.254:34701)
0619 02:50:28.285863 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:28.285865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:28.286009 (+ 144us) server_negotiation.cc:407] Connection header received
0619 02:50:28.286034 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:28.286035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:28.286044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:28.286087 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:28.286192 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45628: BlockingRecv error: recv got EOF from 127.0.0.1:45628 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:50:28.286283 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 31464
I20260619 02:50:28.290310 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 31595
I20260619 02:50:28.293547 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 31726
I20260619 02:50:28.297207 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 31373
2026-06-19T02:50:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (561 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T02:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:28Z Disabled control of system clock
I20260619 02:50:28.321909 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:36847
--webserver_interface=127.21.164.254
--webserver_port=0
--builtin_ntp_servers=127.21.164.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:50:28.400475 31865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:28.400725 31865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:28.400754 31865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:28.401898 31865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:28.401942 31865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:28.401958 31865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:28.401974 31865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:28.401989 31865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:28.403313 31865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:36847
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:36847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:28.403569 31865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:28.403836 31865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:28.408468 31873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.408476 31870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.408497 31871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:28.408492 31865 server_base.cc:1034] running on GCE node
I20260619 02:50:28.408890 31865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:28.409139 31865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:28.410275 31865 hybrid_clock.cc:648] HybridClock initialized: now 1781837428410260 us; error 28 us; skew 500 ppm
I20260619 02:50:28.411500 31865 webserver.cc:460] Webserver started at http://127.21.164.254:42397/ using document root <none> and password file <none>
I20260619 02:50:28.411722 31865 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:28.411772 31865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:28.411899 31865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:28.412993 31865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "f617ea0a058749a59dc23dd87d3ddf60"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "9470011b1be45192b1b12ab51df214c4"
server_key_iv: "43dd2bae807ff1f6357117c16d659ac0"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.413321 31865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "f617ea0a058749a59dc23dd87d3ddf60"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "9470011b1be45192b1b12ab51df214c4"
server_key_iv: "43dd2bae807ff1f6357117c16d659ac0"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.414489 31865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:50:28.415235 31879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.415416 31865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:28.415512 31865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "f617ea0a058749a59dc23dd87d3ddf60"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "9470011b1be45192b1b12ab51df214c4"
server_key_iv: "43dd2bae807ff1f6357117c16d659ac0"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.415581 31865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:28.418920 31865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:28.419179 31865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:28.419298 31865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:28.422830 31865 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:36847
I20260619 02:50:28.422865 31931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:36847 every 8 connection(s)
I20260619 02:50:28.423187 31865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:50:28.423776 31932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:28.425836 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 31865
I20260619 02:50:28.425909 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:50:28.426083 22163 external_mini_cluster.cc:1428] Setting key be5a2b3131ce7bb89b9b009f37d83eee
I20260619 02:50:28.426745 31932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60: Bootstrap starting.
I20260619 02:50:28.427305 31932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:28.427569 31932 log.cc:824] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:28.428246 31932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60: No bootstrap required, opened a new log
I20260619 02:50:28.428776 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.426618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.21.164.254:36847)
0619 02:50:28.426784 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:28.426791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:28.426815 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:50:28.427258 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:28.427264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:28.427306 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:28.427439 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:28.428314 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:28.428318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:28.428318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:28.428323 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:28.428368 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:28.428379 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:28.428447 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:28.428495 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 02:50:28.428548 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260619 02:50:28.430269 31932 raft_consensus.cc:348] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f617ea0a058749a59dc23dd87d3ddf60" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36847 } }
I20260619 02:50:28.430409 31932 raft_consensus.cc:374] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:28.430433 31932 raft_consensus.cc:729] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f617ea0a058749a59dc23dd87d3ddf60, State: Initialized, Role: FOLLOWER
I20260619 02:50:28.430558 31932 consensus_queue.cc:260] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f617ea0a058749a59dc23dd87d3ddf60" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36847 } }
I20260619 02:50:28.430631 31932 raft_consensus.cc:388] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:50:28.430665 31932 raft_consensus.cc:482] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:50:28.430709 31932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:28.431212 31932 raft_consensus.cc:504] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f617ea0a058749a59dc23dd87d3ddf60" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36847 } }
I20260619 02:50:28.431300 31932 leader_election.cc:302] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f617ea0a058749a59dc23dd87d3ddf60; no voters:
I20260619 02:50:28.431463 31932 leader_election.cc:288] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:50:28.431512 31937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:28.431645 31937 raft_consensus.cc:686] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [term 1 LEADER]: Becoming Leader. State: Replica: f617ea0a058749a59dc23dd87d3ddf60, State: Running, Role: LEADER
I20260619 02:50:28.431725 31932 sys_catalog.cc:564] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:28.431739 31937 consensus_queue.cc:237] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f617ea0a058749a59dc23dd87d3ddf60" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36847 } }
I20260619 02:50:28.432121 31938 sys_catalog.cc:455] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f617ea0a058749a59dc23dd87d3ddf60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f617ea0a058749a59dc23dd87d3ddf60" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36847 } } }
I20260619 02:50:28.432126 31939 sys_catalog.cc:455] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f617ea0a058749a59dc23dd87d3ddf60. Latest consensus state: current_term: 1 leader_uuid: "f617ea0a058749a59dc23dd87d3ddf60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f617ea0a058749a59dc23dd87d3ddf60" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36847 } } }
I20260619 02:50:28.432240 31938 sys_catalog.cc:458] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:28.432248 31939 sys_catalog.cc:458] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60 [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:28.432447 31946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:28.432945 31946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:50:28.434405 31946 catalog_manager.cc:1347] Generated new cluster ID: 8b377573aa874b1aaff2e584ab2ae03c
I20260619 02:50:28.434453 31946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:50:28.457942 31946 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:50:28.458129 31946 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:28.461828 31946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f617ea0a058749a59dc23dd87d3ddf60: Generated new TSK 0
I20260619 02:50:28.462007 31946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:50:28.467587 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36847
--builtin_ntp_servers=127.21.164.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:50:28.555850 31956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:28.556061 31956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:28.556092 31956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:28.557276 31956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:28.557317 31956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:28.557353 31956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:50:28.558687 31956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:28.558930 31956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:28.559206 31956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:28.564177 31964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.564244 31962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.564216 31961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:28.564559 31956 server_base.cc:1034] running on GCE node
I20260619 02:50:28.564838 31956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:28.565081 31956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:28.566241 31956 hybrid_clock.cc:648] HybridClock initialized: now 1781837428566225 us; error 31 us; skew 500 ppm
I20260619 02:50:28.567564 31956 webserver.cc:460] Webserver started at http://127.21.164.193:39919/ using document root <none> and password file <none>
I20260619 02:50:28.567787 31956 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:28.567834 31956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:28.567965 31956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:28.569083 31956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "cd89a2c7734c4dd59c3565a02d714f0b"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "444e5e5057644d627f117c9bade63146"
server_key_iv: "2767f29f7a72e14e2869d3a5dc1b2fbe"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.569423 31956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "cd89a2c7734c4dd59c3565a02d714f0b"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "444e5e5057644d627f117c9bade63146"
server_key_iv: "2767f29f7a72e14e2869d3a5dc1b2fbe"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.570750 31956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:28.571471 31970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.571645 31956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:28.571745 31956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "cd89a2c7734c4dd59c3565a02d714f0b"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "444e5e5057644d627f117c9bade63146"
server_key_iv: "2767f29f7a72e14e2869d3a5dc1b2fbe"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.571821 31956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:28.575217 31956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:28.575452 31956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:28.575552 31956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:28.575721 31956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:28.575977 31956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:28.576012 31956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.576032 31956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:28.576052 31956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.581342 31956 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:32889
I20260619 02:50:28.581406 32083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:32889 every 8 connection(s)
I20260619 02:50:28.581707 31956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:50:28.582724 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 31956
I20260619 02:50:28.582809 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:50:28.582986 22163 external_mini_cluster.cc:1428] Setting key 6e64747a7d4e6748553b56b187cc1b6c
I20260619 02:50:28.583983 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36847
--builtin_ntp_servers=127.21.164.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 02:50:28.584849 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.582376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:33481 (local address 127.21.164.254:36847)
0619 02:50:28.582434 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:28.582436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:28.582667 (+ 231us) server_negotiation.cc:407] Connection header received
0619 02:50:28.583246 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:28.583248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:28.583253 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:28.583323 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:28.584571 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:28.584573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:28.584573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:28.584579 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:28.584597 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:28.584602 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:28.584694 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:28.584764 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 02:50:28.584779 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 02:50:28.585018 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.582361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:36847 (local address 127.21.164.193:33481)
0619 02:50:28.582540 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:28.582589 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:28.583131 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:28.583401 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:28.583406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:28.583416 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:28.584409 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:28.584410 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:28.584440 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:28.584477 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:28.584696 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:28.584701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:28.584704 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:28.584760 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:50:28.584815 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260619 02:50:28.585717 32084 heartbeater.cc:347] Connected to a master server at 127.21.164.254:36847
I20260619 02:50:28.585799 32084 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:28.585937 32084 heartbeater.cc:511] Master 127.21.164.254:36847 requested a full tablet report, sending...
I20260619 02:50:28.586429 31896 ts_manager.cc:194] Registered new tserver with Master: cd89a2c7734c4dd59c3565a02d714f0b (127.21.164.193:32889)
W20260619 02:50:28.664022 32087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:28.664248 32087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:28.664271 32087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:28.665567 32087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:28.665616 32087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:28.665644 32087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:50:28.667068 32087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:28.667317 32087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:28.667631 32087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:28.672616 32095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.672607 32093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.672610 32092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:28.672824 32087 server_base.cc:1034] running on GCE node
I20260619 02:50:28.673094 32087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:28.673350 32087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:28.674525 32087 hybrid_clock.cc:648] HybridClock initialized: now 1781837428674508 us; error 32 us; skew 500 ppm
I20260619 02:50:28.675849 32087 webserver.cc:460] Webserver started at http://127.21.164.194:46853/ using document root <none> and password file <none>
I20260619 02:50:28.676092 32087 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:28.676143 32087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:28.676275 32087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:28.677484 32087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "3fdc921ef5da4f78b35c367b3e04b96d"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "def3b815db34f4f831ac8ebbf29c73f9"
server_key_iv: "c5ff712b369c1494ba4dcbb8cacbc639"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.677817 32087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "3fdc921ef5da4f78b35c367b3e04b96d"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "def3b815db34f4f831ac8ebbf29c73f9"
server_key_iv: "c5ff712b369c1494ba4dcbb8cacbc639"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.679123 32087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:50:28.679863 32101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.680029 32087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:50:28.680120 32087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "3fdc921ef5da4f78b35c367b3e04b96d"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "def3b815db34f4f831ac8ebbf29c73f9"
server_key_iv: "c5ff712b369c1494ba4dcbb8cacbc639"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.680184 32087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:28.683648 32087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:28.683904 32087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:28.684021 32087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:28.684207 32087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:28.684509 32087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:28.684542 32087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.684569 32087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:28.684592 32087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.689925 32087 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:43397
I20260619 02:50:28.689970 32214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:43397 every 8 connection(s)
I20260619 02:50:28.690279 32087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:50:28.693259 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.691071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:38551 (local address 127.21.164.254:36847)
0619 02:50:28.691148 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:28.691150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:28.691393 (+ 243us) server_negotiation.cc:407] Connection header received
0619 02:50:28.691865 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:28.691866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:28.691872 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:28.691937 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:28.693072 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:28.693074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:28.693074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:28.693077 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:28.693091 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:28.693095 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:28.693143 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:28.693192 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 02:50:28.693203 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:50:28.693442 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.691042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:36847 (local address 127.21.164.194:38551)
0619 02:50:28.691218 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:28.691261 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:28.691761 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:28.691994 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:28.691998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:28.692007 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:28.692932 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:28.692933 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:28.692964 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:28.692995 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:28.693140 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:28.693142 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:28.693145 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:28.693194 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 02:50:28.693242 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260619 02:50:28.693912 32215 heartbeater.cc:347] Connected to a master server at 127.21.164.254:36847
I20260619 02:50:28.693990 32215 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:28.694128 32215 heartbeater.cc:511] Master 127.21.164.254:36847 requested a full tablet report, sending...
I20260619 02:50:28.694449 31896 ts_manager.cc:194] Registered new tserver with Master: 3fdc921ef5da4f78b35c367b3e04b96d (127.21.164.194:43397)
I20260619 02:50:28.699133 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 32087
I20260619 02:50:28.699242 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:50:28.699434 22163 external_mini_cluster.cc:1428] Setting key f4d9923ff11eded21b86a491d8b659d3
I20260619 02:50:28.700292 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36847
--builtin_ntp_servers=127.21.164.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:50:28.779830 32218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:28.780053 32218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:28.780076 32218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:28.781287 32218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:28.781329 32218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:28.781355 32218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:50:28.782682 32218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:28.782939 32218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:28.783239 32218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:28.787962 32223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.788004 32224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:28.787988 32226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:28.788120 32218 server_base.cc:1034] running on GCE node
I20260619 02:50:28.788395 32218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:28.788623 32218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:28.789790 32218 hybrid_clock.cc:648] HybridClock initialized: now 1781837428789771 us; error 36 us; skew 500 ppm
I20260619 02:50:28.790972 32218 webserver.cc:460] Webserver started at http://127.21.164.195:36469/ using document root <none> and password file <none>
I20260619 02:50:28.791193 32218 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:28.791237 32218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:28.791371 32218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:28.792420 32218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "68908a3796944281adb77b0a7a448de6"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "b02b57241f112a4591fe6236d46ac5e3"
server_key_iv: "6122331568a3758fc3e78992f468efd3"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.792771 32218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "68908a3796944281adb77b0a7a448de6"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "b02b57241f112a4591fe6236d46ac5e3"
server_key_iv: "6122331568a3758fc3e78992f468efd3"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.793990 32218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:50:28.794718 32232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.794934 32218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:50:28.795023 32218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "68908a3796944281adb77b0a7a448de6"
format_stamp: "Formatted at 2026-06-19 02:50:28 on dist-test-slave-jt0t"
server_key: "b02b57241f112a4591fe6236d46ac5e3"
server_key_iv: "6122331568a3758fc3e78992f468efd3"
server_key_version: "encryptionkey@0"
I20260619 02:50:28.795078 32218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:28.798297 32218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:28.798512 32218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:28.798599 32218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:28.798740 32218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:28.798969 32218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:28.798996 32218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.799014 32218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:28.799026 32218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:28.804072 32218 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:40635
I20260619 02:50:28.804136 32345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:40635 every 8 connection(s)
I20260619 02:50:28.804360 32218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:50:28.804590 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 32218
I20260619 02:50:28.804718 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:50:28.804939 22163 external_mini_cluster.cc:1428] Setting key 9a017d0e353b006fbbd4481cfe40efc9
I20260619 02:50:28.807427 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.805121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:50717 (local address 127.21.164.254:36847)
0619 02:50:28.805185 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:28.805188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:28.805539 (+ 351us) server_negotiation.cc:407] Connection header received
0619 02:50:28.806032 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:28.806033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:28.806039 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:28.806080 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:28.807206 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:28.807207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:28.807208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:50:28.807212 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:28.807228 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:50:28.807232 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:28.807281 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:28.807348 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 02:50:28.807361 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:50:28.807617 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:28.805138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:36847 (local address 127.21.164.195:50717)
0619 02:50:28.805413 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:28.805454 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:28.805942 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:28.806150 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:28.806154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:28.806165 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:28.807071 (+ 906us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:50:28.807072 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:28.807100 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:50:28.807128 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:28.807286 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:28.807291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:28.807294 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:28.807360 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:50:28.807419 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":95,"threads_started":1}
I20260619 02:50:28.807983 32346 heartbeater.cc:347] Connected to a master server at 127.21.164.254:36847
I20260619 02:50:28.808065 32346 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:28.808178 32346 heartbeater.cc:511] Master 127.21.164.254:36847 requested a full tablet report, sending...
I20260619 02:50:28.808465 31896 ts_manager.cc:194] Registered new tserver with Master: 68908a3796944281adb77b0a7a448de6 (127.21.164.195:40635)
I20260619 02:50:28.816491 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:50:28.818336 31935 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:50:28.817864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.21.164.254:36847)
0619 02:50:28.817926 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:28.817928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:28.818021 (+ 93us) server_negotiation.cc:407] Connection header received
0619 02:50:28.818083 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:28.818084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:28.818089 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:28.818136 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:28.818272 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41752: BlockingRecv error: recv got EOF from 127.0.0.1:41752 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:50:28.818619 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 31956
I20260619 02:50:28.822698 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 32087
I20260619 02:50:28.826676 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 32218
I20260619 02:50:28.830075 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 31865
2026-06-19T02:50:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (532 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:50:28 dist-test-slave-jt0t krb5kdc[32356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:50:28 dist-test-slave-jt0t krb5kdc[32356](info): set up 2 sockets
Jun 19 02:50:28 dist-test-slave-jt0t krb5kdc[32356](info): commencing operation
krb5kdc: starting...
W20260619 02:50:29.777315 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:50:29 dist-test-slave-jt0t krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:50:29.891043 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45867
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:45867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:29.970949 32369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:29.971175 32369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:29.971206 32369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:29.972410 32369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:50:29.972457 32369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:29.972476 32369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:29.972492 32369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:50:29.972506 32369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:50:29.974133 32369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:45867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:29.974422 32369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:29.974640 32369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:29.979560 32375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:29.979600 32369 server_base.cc:1034] running on GCE node
W20260619 02:50:29.979570 32377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:29.979548 32374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:29.979970 32369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:29.980378 32369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:29.981516 32369 hybrid_clock.cc:648] HybridClock initialized: now 1781837429981508 us; error 36 us; skew 500 ppm
Jun 19 02:50:29 dist-test-slave-jt0t krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837429, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:29.983916 32369 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:50:29.984350 32369 webserver.cc:460] Webserver started at http://127.21.164.254:38767/ using document root <none> and password file <none>
I20260619 02:50:29.984552 32369 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:29.984592 32369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:29.984722 32369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:29.985754 32369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "0c4e0fe487fa4720980b20e0d2c24dce"
format_stamp: "Formatted at 2026-06-19 02:50:29 on dist-test-slave-jt0t"
server_key: "50076d420dc717f7f62f49bc747a0752"
server_key_iv: "b12927fbe52ae525c48aedaba816fed4"
server_key_version: "encryptionkey@0"
I20260619 02:50:29.986034 32369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "0c4e0fe487fa4720980b20e0d2c24dce"
format_stamp: "Formatted at 2026-06-19 02:50:29 on dist-test-slave-jt0t"
server_key: "50076d420dc717f7f62f49bc747a0752"
server_key_iv: "b12927fbe52ae525c48aedaba816fed4"
server_key_version: "encryptionkey@0"
I20260619 02:50:29.987180 32369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:50:29.987818 32384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:29.987980 32369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:29.988057 32369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "0c4e0fe487fa4720980b20e0d2c24dce"
format_stamp: "Formatted at 2026-06-19 02:50:29 on dist-test-slave-jt0t"
server_key: "50076d420dc717f7f62f49bc747a0752"
server_key_iv: "b12927fbe52ae525c48aedaba816fed4"
server_key_version: "encryptionkey@0"
I20260619 02:50:29.988113 32369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:30.003371 32369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:30.005182 32369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:30.005293 32369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:30.008903 32369 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45867
I20260619 02:50:30.008951 32436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45867 every 8 connection(s)
I20260619 02:50:30.009276 32369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:50:30.010208 32437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:30.012517 32437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce: Bootstrap starting.
I20260619 02:50:30.013125 32437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:30.013372 32437 log.cc:824] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:30.014020 32437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce: No bootstrap required, opened a new log
I20260619 02:50:30.015642 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 32369
I20260619 02:50:30.015574 32437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4e0fe487fa4720980b20e0d2c24dce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45867 } }
I20260619 02:50:30.015727 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:50:30.015714 32437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:30.015748 32437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4e0fe487fa4720980b20e0d2c24dce, State: Initialized, Role: FOLLOWER
I20260619 02:50:30.015928 22163 external_mini_cluster.cc:1428] Setting key 7a2d476827ed3ddddc0563965e502d78
I20260619 02:50:30.015889 32437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4e0fe487fa4720980b20e0d2c24dce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45867 } }
I20260619 02:50:30.015964 32437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:50:30.015995 32437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:50:30.016044 32437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:30.016767 32437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4e0fe487fa4720980b20e0d2c24dce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45867 } }
I20260619 02:50:30.016871 32437 leader_election.cc:302] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c4e0fe487fa4720980b20e0d2c24dce; no voters:
I20260619 02:50:30.017035 32437 leader_election.cc:288] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:50:30.017074 32442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:30.017213 32442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [term 1 LEADER]: Becoming Leader. State: Replica: 0c4e0fe487fa4720980b20e0d2c24dce, State: Running, Role: LEADER
I20260619 02:50:30.017316 32437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:50:30.017316 32442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4e0fe487fa4720980b20e0d2c24dce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45867 } }
I20260619 02:50:30.017905 32444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c4e0fe487fa4720980b20e0d2c24dce. Latest consensus state: current_term: 1 leader_uuid: "0c4e0fe487fa4720980b20e0d2c24dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4e0fe487fa4720980b20e0d2c24dce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45867 } } }
I20260619 02:50:30.017922 32443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c4e0fe487fa4720980b20e0d2c24dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4e0fe487fa4720980b20e0d2c24dce" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45867 } } }
I20260619 02:50:30.018003 32444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [sys.catalog]: This master's current role is: LEADER
I20260619 02:50:30.018018 32443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce [sys.catalog]: This master's current role is: LEADER
W20260619 02:50:30.019188 32457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:50:30.019245 32457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:50:30.019340 32451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:50:30.019568 32451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:50:30.021224 32451 catalog_manager.cc:1347] Generated new cluster ID: d745e468cd5842febb5e9780237a44f7
I20260619 02:50:30.021267 32451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:50:30 dist-test-slave-jt0t krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:50:30.028353 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.016412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.21.164.254:45867)
0619 02:50:30.016927 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:30.016939 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:30.016976 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:50:30.017102 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:30.017109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:30.017171 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:30.018074 (+ 903us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:30.018182 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.019610 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.020399 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.020961 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.021065 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.023849 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:30.023853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:30.023853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:30.023873 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:30.026580 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:30.027003 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:30.027006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:30.027007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:30.027156 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:30.027370 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:30.027371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:30.027371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:30.027686 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:30.027823 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:30.027958 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 02:50:30.028106 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":390,"threads_started":1}
I20260619 02:50:30.031903 32451 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:50:30.032403 32451 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:50:30.036684 32451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce: Generated new TSK 0
I20260619 02:50:30.036875 32451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:50:30.063493 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45867
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:30.144218 32463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:30.144431 32463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:30.144454 32463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:30.145632 32463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:30.145669 32463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:30.145695 32463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:50:30.147130 32463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:30.147331 32463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:30.147548 32463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:30.152428 32469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:30.152484 32468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:30.152547 32471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:30.152642 32463 server_base.cc:1034] running on GCE node
I20260619 02:50:30.152860 32463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:30.153067 32463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:30.154199 32463 hybrid_clock.cc:648] HybridClock initialized: now 1781837430154184 us; error 31 us; skew 500 ppm
Jun 19 02:50:30 dist-test-slave-jt0t krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:30.156828 32463 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:50:30.157243 32463 webserver.cc:460] Webserver started at http://127.21.164.193:35313/ using document root <none> and password file <none>
I20260619 02:50:30.157447 32463 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:30.157495 32463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:30.157624 32463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:30.158641 32463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "8cb121351d8b4e98a80d66f2448f5297"
format_stamp: "Formatted at 2026-06-19 02:50:30 on dist-test-slave-jt0t"
server_key: "8a1362feabce670f04075605a100e810"
server_key_iv: "ef2175da4d921a9d7123643c99389889"
server_key_version: "encryptionkey@0"
I20260619 02:50:30.158967 32463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "8cb121351d8b4e98a80d66f2448f5297"
format_stamp: "Formatted at 2026-06-19 02:50:30 on dist-test-slave-jt0t"
server_key: "8a1362feabce670f04075605a100e810"
server_key_iv: "ef2175da4d921a9d7123643c99389889"
server_key_version: "encryptionkey@0"
I20260619 02:50:30.160195 32463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 02:50:30.160982 32478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:30.161187 32463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:30.161278 32463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "8cb121351d8b4e98a80d66f2448f5297"
format_stamp: "Formatted at 2026-06-19 02:50:30 on dist-test-slave-jt0t"
server_key: "8a1362feabce670f04075605a100e810"
server_key_iv: "ef2175da4d921a9d7123643c99389889"
server_key_version: "encryptionkey@0"
I20260619 02:50:30.161338 32463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:30.191956 32463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:30.194135 32463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:30.194262 32463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:30.194483 32463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:30.194808 32463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:30.194849 32463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:30.194877 32463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:30.194895 32463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:30.200289 32463 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:38113
I20260619 02:50:30.200336 32591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:38113 every 8 connection(s)
I20260619 02:50:30.200670 32463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:50:30 dist-test-slave-jt0t krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:50:30.208544 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 32463
I20260619 02:50:30.208678 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:50:30.208889 22163 external_mini_cluster.cc:1428] Setting key a03948d481e44d252e2d7c2f8b2ac23a
I20260619 02:50:30.209993 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.201404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:48107 (local address 127.21.164.254:45867)
0619 02:50:30.201458 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:30.201460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:30.201693 (+ 233us) server_negotiation.cc:407] Connection header received
0619 02:50:30.202199 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:30.202200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:30.202213 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:30.202265 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:30.203227 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.203739 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.204725 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.204935 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.204991 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.207075 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:30.207077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:30.207077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:30.207085 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:30.208376 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:30.208892 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:30.208893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:30.208894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:30.208954 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:30.209245 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:30.209246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:30.209246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:30.209369 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:30.209456 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:30.209808 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 02:50:30.209904 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:50:30.210168 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.201381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45867 (local address 127.21.164.193:48107)
0619 02:50:30.201545 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:30.201605 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:30.202099 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:30.202345 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:30.202348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:30.202924 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:30.203125 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.203130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.203823 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.203824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.204627 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.204630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.205003 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.205004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.205135 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.205644 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:30.205653 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:30.206977 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:30.208460 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:30.208466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:30.208469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:30.208824 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:30.209029 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:30.209030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:30.209030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:30.209186 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:30.209554 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:30.209560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:30.209684 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 02:50:30.209776 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:50:30.209948 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260619 02:50:30.210614 32592 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45867
I20260619 02:50:30.210724 32592 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:30.210902 32592 heartbeater.cc:511] Master 127.21.164.254:45867 requested a full tablet report, sending...
I20260619 02:50:30.211421 32401 ts_manager.cc:194] Registered new tserver with Master: 8cb121351d8b4e98a80d66f2448f5297 (127.21.164.193:38113)
I20260619 02:50:30.212108 32401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:48107
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:50:30.233376 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45867
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:30.313220 32597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:30.313429 32597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:30.313452 32597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:30.314579 32597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:30.314615 32597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:30.314639 32597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:50:30.316040 32597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:30.316239 32597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:30.316418 32597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:30.321394 32605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:30.321380 32602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:30.321375 32603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:30.321635 32597 server_base.cc:1034] running on GCE node
I20260619 02:50:30.321846 32597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:30.322069 32597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:30.323236 32597 hybrid_clock.cc:648] HybridClock initialized: now 1781837430323221 us; error 30 us; skew 500 ppm
Jun 19 02:50:30 dist-test-slave-jt0t krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:30.325690 32597 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:50:30.326135 32597 webserver.cc:460] Webserver started at http://127.21.164.194:43431/ using document root <none> and password file <none>
I20260619 02:50:30.326351 32597 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:30.326399 32597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:30.326535 32597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:30.327576 32597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "e374899e2bd64b98a90e3aef2e23a951"
format_stamp: "Formatted at 2026-06-19 02:50:30 on dist-test-slave-jt0t"
server_key: "f3a219bd637d0e291773030180fef327"
server_key_iv: "7a6c54a65dc749442bff6f9091b87e1e"
server_key_version: "encryptionkey@0"
I20260619 02:50:30.327905 32597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "e374899e2bd64b98a90e3aef2e23a951"
format_stamp: "Formatted at 2026-06-19 02:50:30 on dist-test-slave-jt0t"
server_key: "f3a219bd637d0e291773030180fef327"
server_key_iv: "7a6c54a65dc749442bff6f9091b87e1e"
server_key_version: "encryptionkey@0"
I20260619 02:50:30.329233 32597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 02:50:30.330159 32612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:30.330319 32597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:50:30.330399 32597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "e374899e2bd64b98a90e3aef2e23a951"
format_stamp: "Formatted at 2026-06-19 02:50:30 on dist-test-slave-jt0t"
server_key: "f3a219bd637d0e291773030180fef327"
server_key_iv: "7a6c54a65dc749442bff6f9091b87e1e"
server_key_version: "encryptionkey@0"
I20260619 02:50:30.330469 32597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:30.344100 32597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:30.346033 32597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:30.346176 32597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:30.346391 32597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:30.346702 32597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:30.346735 32597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:30.346765 32597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:30.346787 32597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:30.352790 32597 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:42137
I20260619 02:50:30.352846 32725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:42137 every 8 connection(s)
I20260619 02:50:30.353152 32597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:50:30.357923 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 32597
I20260619 02:50:30.358038 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:50:30.358210 22163 external_mini_cluster.cc:1428] Setting key d9883397495724033d59292baad4d90d
Jun 19 02:50:30 dist-test-slave-jt0t krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:50:30.362291 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.353831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:58569 (local address 127.21.164.254:45867)
0619 02:50:30.353885 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:30.353887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:30.354085 (+ 198us) server_negotiation.cc:407] Connection header received
0619 02:50:30.354600 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:30.354601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:30.354614 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:30.354664 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:30.355622 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.356111 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.357120 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.357324 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.357378 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.359679 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:30.359680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:30.359681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:30.359688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:30.361043 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:30.361415 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:30.361416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:30.361416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:30.361467 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:30.361694 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:30.361696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:30.361696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:30.361802 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:30.361857 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:30.362122 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 02:50:30.362201 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:50:30.362483 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.353813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45867 (local address 127.21.164.194:58569)
0619 02:50:30.353961 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:30.353997 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:30.354503 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:30.354737 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:30.354741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:30.355325 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:30.355530 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.355536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.356193 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.356194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.357042 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.357045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.357390 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.357391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.357505 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.358055 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:30.358065 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:30.359597 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:30.361115 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:30.361122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:30.361125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:30.361351 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:30.361520 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:30.361521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:30.361521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:30.361643 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:30.361853 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:30.361859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:30.362000 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 02:50:30.362097 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 02:50:30.362261 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260619 02:50:30.362943 32726 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45867
I20260619 02:50:30.363034 32726 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:30.363204 32726 heartbeater.cc:511] Master 127.21.164.254:45867 requested a full tablet report, sending...
I20260619 02:50:30.363585 32401 ts_manager.cc:194] Registered new tserver with Master: e374899e2bd64b98a90e3aef2e23a951 (127.21.164.194:42137)
I20260619 02:50:30.364025 32401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:58569
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:50:30.382788 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45867
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:45055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:50:30.463994 32731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:50:30.464233 32731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:50:30.464257 32731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:50:30.465421 32731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:50:30.465467 32731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:50:30.465492 32731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:50:30.466869 32731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:50:30.467097 32731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:50:30.467294 32731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:50:30.472000 32736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:50:30.472010 32739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:30.472064 32731 server_base.cc:1034] running on GCE node
W20260619 02:50:30.472005 32737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:50:30.472295 32731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:50:30.472500 32731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:50:30.473654 32731 hybrid_clock.cc:648] HybridClock initialized: now 1781837430473622 us; error 41 us; skew 500 ppm
Jun 19 02:50:30 dist-test-slave-jt0t krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:50:30.476100 32731 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:50:30.476511 32731 webserver.cc:460] Webserver started at http://127.21.164.195:46729/ using document root <none> and password file <none>
I20260619 02:50:30.476727 32731 fs_manager.cc:359] Metadata directory not provided
I20260619 02:50:30.476773 32731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:50:30.476864 32731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:50:30.477857 32731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "d9c3c29089044903aa11c345b15b44af"
format_stamp: "Formatted at 2026-06-19 02:50:30 on dist-test-slave-jt0t"
server_key: "1eca73a89354ffc82c8f453d8b288288"
server_key_iv: "b143a1ec2b07bca7b98fe05dfb9d4567"
server_key_version: "encryptionkey@0"
I20260619 02:50:30.478165 32731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "d9c3c29089044903aa11c345b15b44af"
format_stamp: "Formatted at 2026-06-19 02:50:30 on dist-test-slave-jt0t"
server_key: "1eca73a89354ffc82c8f453d8b288288"
server_key_iv: "b143a1ec2b07bca7b98fe05dfb9d4567"
server_key_version: "encryptionkey@0"
I20260619 02:50:30.479319 32731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:50:30.479952 32746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:30.480080 32731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:50:30.480154 32731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "d9c3c29089044903aa11c345b15b44af"
format_stamp: "Formatted at 2026-06-19 02:50:30 on dist-test-slave-jt0t"
server_key: "1eca73a89354ffc82c8f453d8b288288"
server_key_iv: "b143a1ec2b07bca7b98fe05dfb9d4567"
server_key_version: "encryptionkey@0"
I20260619 02:50:30.480221 32731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:50:30.489152 32731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:50:30.491024 32731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:50:30.491151 32731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:50:30.491394 32731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:50:30.491722 32731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:50:30.491758 32731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:30.491786 32731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:50:30.491812 32731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:50:30.497287 32731 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:44277
I20260619 02:50:30.497326 391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:44277 every 8 connection(s)
I20260619 02:50:30.497644 32731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:50:30 dist-test-slave-jt0t krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837430, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:50:30.506067 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.498286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:50165 (local address 127.21.164.254:45867)
0619 02:50:30.498342 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:30.498344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:30.498527 (+ 183us) server_negotiation.cc:407] Connection header received
0619 02:50:30.498981 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:30.498982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:30.498992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:30.499037 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:30.499919 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.500395 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.501431 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.501642 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.501697 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.503745 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:30.503746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:30.503747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:30.503754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:30.505030 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:30.505355 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:30.505356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:30.505356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:30.505409 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:30.505583 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:30.505583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:30.505583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:30.505685 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:30.505731 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:30.505941 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 02:50:30.506000 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:50:30.506270 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.498268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45867 (local address 127.21.164.195:50165)
0619 02:50:30.498415 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:30.498450 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:30.498907 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:30.499108 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:30.499111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:30.499659 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:50:30.499854 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.499859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.500492 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.500493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.501352 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.501354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.501735 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.501737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.501858 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.502295 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:50:30.502303 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:50:30.503650 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:50:30.505095 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:30.505099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:30.505101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:30.505302 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:30.505451 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:50:30.505452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:50:30.505452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:50:30.505539 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:50:30.505729 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:50:30.505732 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:50:30.505845 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 02:50:30.505923 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:50:30.506041 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260619 02:50:30.506809 392 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45867
I20260619 02:50:30.506916 392 heartbeater.cc:464] Registering TS with master...
I20260619 02:50:30.507109 392 heartbeater.cc:511] Master 127.21.164.254:45867 requested a full tablet report, sending...
I20260619 02:50:30.507560 32401 ts_manager.cc:194] Registered new tserver with Master: d9c3c29089044903aa11c345b15b44af (127.21.164.195:44277)
I20260619 02:50:30.507653 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 32731
I20260619 02:50:30.507746 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:50:30.507908 22163 external_mini_cluster.cc:1428] Setting key 34e05982b97ed5e206a56f17a102a8a2
I20260619 02:50:30.508057 32401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:50165
I20260619 02:50:30.508913 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:50:30.517622 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.510638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.21.164.254:45867)
0619 02:50:30.510683 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:30.510685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:30.510790 (+ 105us) server_negotiation.cc:407] Connection header received
0619 02:50:30.510885 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:30.510885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:30.510897 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:30.510962 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:50:30.511639 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.512242 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.513253 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.513571 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.513640 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.514990 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:50:30.514992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:50:30.514992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:50:30.514999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:50:30.516560 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:30.516907 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:30.516908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:30.516908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:30.516962 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:50:30.517092 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:50:30.517092 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:50:30.517093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:50:30.517272 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:50:30.517343 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:30.517464 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 02:50:30.517530 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:50:30.519253 32401 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:50:30.520761 32401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:50:30.526997 401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.522698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:42137 (local address 127.0.0.1:37122)
0619 02:50:30.523455 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:30.523462 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:30.523504 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:30.523778 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:30.523779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:30.523781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:30.523825 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.523826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.524964 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.524965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.525844 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.525846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.526679 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.526680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.526749 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.526760 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 02:50:30.526820 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:50:30.526932 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":53,"thread_start_us":65,"threads_started":1}
I20260619 02:50:30.526991 404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.522924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.21.164.194:42137)
0619 02:50:30.523090 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:30.523095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:30.523567 (+ 472us) server_negotiation.cc:407] Connection header received
0619 02:50:30.523612 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:30.523615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:30.523631 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:30.523704 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:30.523895 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.524883 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.525914 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.526607 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.526684 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.526740 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:30.526854 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 02:50:30.526905 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260619 02:50:30.527283 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.522972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.21.164.193:38113)
0619 02:50:30.523164 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:30.523167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:30.523240 (+ 73us) server_negotiation.cc:407] Connection header received
0619 02:50:30.523321 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:30.523324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:30.523340 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:30.523411 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:30.523547 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.524642 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.526057 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.526727 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.526783 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.526837 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:30.527161 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 02:50:30.527205 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260619 02:50:30.527096 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.522698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:38113 (local address 127.0.0.1:47500)
0619 02:50:30.523135 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:30.523177 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:30.523249 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:30.523420 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:30.523423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:30.523425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:30.523483 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.523485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.524771 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.524772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.525990 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.525993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.526869 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.526869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.526932 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.526935 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:30.526981 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:50:30.527022 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":71,"threads_started":1}
I20260619 02:50:30.528023 32660 tablet_service.cc:1467] Processing CreateTablet for tablet 7d478f7a7ca947c7929a6f19cd06ec5a (DEFAULT_TABLE table=test-table [id=472e65a660cc4f3394e57b7ada217bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:30.528033 32526 tablet_service.cc:1467] Processing CreateTablet for tablet 7d478f7a7ca947c7929a6f19cd06ec5a (DEFAULT_TABLE table=test-table [id=472e65a660cc4f3394e57b7ada217bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:30.528343 32660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d478f7a7ca947c7929a6f19cd06ec5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:30.528357 32526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d478f7a7ca947c7929a6f19cd06ec5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:30.529040 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.522701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:44277 (local address 127.0.0.1:49008)
0619 02:50:30.523340 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:30.523349 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:30.523395 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:30.523792 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:30.523792 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:30.523794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:30.523823 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.523825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.526775 (+ 2950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.526776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.527846 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.527849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.528811 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.528812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.528873 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.528876 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:30.528929 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 02:50:30.528976 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":63,"threads_started":1}
I20260619 02:50:30.529315 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.522873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49008 (local address 127.21.164.195:44277)
0619 02:50:30.523060 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:30.523066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:30.523615 (+ 549us) server_negotiation.cc:407] Connection header received
0619 02:50:30.523654 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:30.523657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:30.523670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:30.523732 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:30.524722 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.525812 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.527932 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.528625 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.528721 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.528776 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:30.528958 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 02:50:30.529003 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
I20260619 02:50:30.531070 406 tablet_bootstrap.cc:492] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297: Bootstrap starting.
I20260619 02:50:30.531729 406 tablet_bootstrap.cc:654] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:30.531692 326 tablet_service.cc:1467] Processing CreateTablet for tablet 7d478f7a7ca947c7929a6f19cd06ec5a (DEFAULT_TABLE table=test-table [id=472e65a660cc4f3394e57b7ada217bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:50:30.531955 326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d478f7a7ca947c7929a6f19cd06ec5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:50:30.532222 406 log.cc:824] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:30.532495 407 tablet_bootstrap.cc:492] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: Bootstrap starting.
I20260619 02:50:30.532929 406 tablet_bootstrap.cc:492] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297: No bootstrap required, opened a new log
I20260619 02:50:30.533001 406 ts_tablet_manager.cc:1397] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:50:30.533041 407 tablet_bootstrap.cc:654] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:30.533502 407 log.cc:824] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:30.534170 407 tablet_bootstrap.cc:492] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: No bootstrap required, opened a new log
I20260619 02:50:30.534235 407 ts_tablet_manager.cc:1397] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:50:30.534880 406 raft_consensus.cc:348] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb121351d8b4e98a80d66f2448f5297" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38113 } } peers { permanent_uuid: "d9c3c29089044903aa11c345b15b44af" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 44277 } } peers { permanent_uuid: "e374899e2bd64b98a90e3aef2e23a951" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 42137 } }
I20260619 02:50:30.535053 406 raft_consensus.cc:374] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:30.535094 406 raft_consensus.cc:729] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb121351d8b4e98a80d66f2448f5297, State: Initialized, Role: FOLLOWER
I20260619 02:50:30.535033 410 tablet_bootstrap.cc:492] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af: Bootstrap starting.
I20260619 02:50:30.535219 406 consensus_queue.cc:260] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb121351d8b4e98a80d66f2448f5297" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38113 } } peers { permanent_uuid: "d9c3c29089044903aa11c345b15b44af" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 44277 } } peers { permanent_uuid: "e374899e2bd64b98a90e3aef2e23a951" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 42137 } }
I20260619 02:50:30.535511 406 ts_tablet_manager.cc:1428] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 02:50:30.535646 32592 heartbeater.cc:503] Master 127.21.164.254:45867 was elected leader, sending a full tablet report...
I20260619 02:50:30.535966 410 tablet_bootstrap.cc:654] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af: Neither blocks nor log segments found. Creating new log.
I20260619 02:50:30.535908 407 raft_consensus.cc:348] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb121351d8b4e98a80d66f2448f5297" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38113 } } peers { permanent_uuid: "d9c3c29089044903aa11c345b15b44af" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 44277 } } peers { permanent_uuid: "e374899e2bd64b98a90e3aef2e23a951" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 42137 } }
I20260619 02:50:30.536065 407 raft_consensus.cc:374] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:30.536111 407 raft_consensus.cc:729] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e374899e2bd64b98a90e3aef2e23a951, State: Initialized, Role: FOLLOWER
I20260619 02:50:30.536228 407 consensus_queue.cc:260] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb121351d8b4e98a80d66f2448f5297" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38113 } } peers { permanent_uuid: "d9c3c29089044903aa11c345b15b44af" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 44277 } } peers { permanent_uuid: "e374899e2bd64b98a90e3aef2e23a951" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 42137 } }
I20260619 02:50:30.536504 32726 heartbeater.cc:503] Master 127.21.164.254:45867 was elected leader, sending a full tablet report...
I20260619 02:50:30.536490 410 log.cc:824] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af: Log is configured to *not* fsync() on all Append() calls
I20260619 02:50:30.536814 407 ts_tablet_manager.cc:1428] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:50:30.537289 410 tablet_bootstrap.cc:492] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af: No bootstrap required, opened a new log
I20260619 02:50:30.537355 410 ts_tablet_manager.cc:1397] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:50:30.538666 410 raft_consensus.cc:348] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb121351d8b4e98a80d66f2448f5297" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38113 } } peers { permanent_uuid: "d9c3c29089044903aa11c345b15b44af" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 44277 } } peers { permanent_uuid: "e374899e2bd64b98a90e3aef2e23a951" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 42137 } }
I20260619 02:50:30.538795 410 raft_consensus.cc:374] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:50:30.538825 410 raft_consensus.cc:729] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c3c29089044903aa11c345b15b44af, State: Initialized, Role: FOLLOWER
I20260619 02:50:30.538935 410 consensus_queue.cc:260] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb121351d8b4e98a80d66f2448f5297" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38113 } } peers { permanent_uuid: "d9c3c29089044903aa11c345b15b44af" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 44277 } } peers { permanent_uuid: "e374899e2bd64b98a90e3aef2e23a951" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 42137 } }
I20260619 02:50:30.539167 410 ts_tablet_manager.cc:1428] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 02:50:30.539204 392 heartbeater.cc:503] Master 127.21.164.254:45867 was elected leader, sending a full tablet report...
I20260619 02:50:30.602494 414 raft_consensus.cc:482] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:50:30.602622 414 raft_consensus.cc:504] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb121351d8b4e98a80d66f2448f5297" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38113 } } peers { permanent_uuid: "d9c3c29089044903aa11c345b15b44af" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 44277 } } peers { permanent_uuid: "e374899e2bd64b98a90e3aef2e23a951" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 42137 } }
I20260619 02:50:30.602962 414 leader_election.cc:288] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cb121351d8b4e98a80d66f2448f5297 (127.21.164.193:38113), e374899e2bd64b98a90e3aef2e23a951 (127.21.164.194:42137)
W20260619 02:50:30.603667 32727 tablet.cc:2369] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:50:30.605914 404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.603267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:50831 (local address 127.21.164.194:42137)
0619 02:50:30.603314 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:30.603316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:30.603325 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:50:30.603352 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:30.603353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:30.603360 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:30.603411 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:30.603564 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.604140 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.605163 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.605588 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.605643 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.605651 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:30.605809 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 02:50:30.605846 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:50:30.605908 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.603136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:42137 (local address 127.21.164.195:50831)
0619 02:50:30.603189 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:30.603201 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:30.603247 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:30.603438 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:30.603439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:30.603441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:30.603504 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.603506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.604218 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.604219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.605078 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.605080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.605665 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.605666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.605742 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.605745 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:50:30.605793 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:50:30.605830 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":33}
I20260619 02:50:30.605966 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.603248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:48083 (local address 127.21.164.193:38113)
0619 02:50:30.603295 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:30.603302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:30.603560 (+ 258us) server_negotiation.cc:407] Connection header received
0619 02:50:30.603583 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:30.603584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:30.603591 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:30.603629 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:50:30.603803 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.604321 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.605284 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.605670 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.605739 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.605747 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:30.605871 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 02:50:30.605907 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:50:30.606077 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.603137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:38113 (local address 127.21.164.195:48083)
0619 02:50:30.603358 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:30.603365 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 02:50:30.603412 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:50:30.603646 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:50:30.603647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:50:30.603649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:50:30.603706 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.603707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.604399 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.604400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.605199 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:50:30.605202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.605744 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.605745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:50:30.605804 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.605808 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:50:30.605858 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 02:50:30.605906 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":19,"thread_start_us":121,"threads_started":1}
I20260619 02:50:30.606292 32680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d478f7a7ca947c7929a6f19cd06ec5a" candidate_uuid: "d9c3c29089044903aa11c345b15b44af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e374899e2bd64b98a90e3aef2e23a951" is_pre_election: true
I20260619 02:50:30.606452 32680 raft_consensus.cc:2454] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9c3c29089044903aa11c345b15b44af in term 0.
I20260619 02:50:30.606374 32546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d478f7a7ca947c7929a6f19cd06ec5a" candidate_uuid: "d9c3c29089044903aa11c345b15b44af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb121351d8b4e98a80d66f2448f5297" is_pre_election: true
I20260619 02:50:30.606591 32546 raft_consensus.cc:2454] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9c3c29089044903aa11c345b15b44af in term 0.
I20260619 02:50:30.606684 32749 leader_election.cc:302] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9c3c29089044903aa11c345b15b44af, e374899e2bd64b98a90e3aef2e23a951; no voters:
I20260619 02:50:30.607666 414 raft_consensus.cc:2781] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:50:30.607724 414 raft_consensus.cc:482] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:50:30.607739 414 raft_consensus.cc:3037] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:30.608430 414 raft_consensus.cc:504] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb121351d8b4e98a80d66f2448f5297" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38113 } } peers { permanent_uuid: "d9c3c29089044903aa11c345b15b44af" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 44277 } } peers { permanent_uuid: "e374899e2bd64b98a90e3aef2e23a951" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 42137 } }
I20260619 02:50:30.608552 414 leader_election.cc:288] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [CANDIDATE]: Term 1 election: Requested vote from peers 8cb121351d8b4e98a80d66f2448f5297 (127.21.164.193:38113), e374899e2bd64b98a90e3aef2e23a951 (127.21.164.194:42137)
I20260619 02:50:30.608716 32546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d478f7a7ca947c7929a6f19cd06ec5a" candidate_uuid: "d9c3c29089044903aa11c345b15b44af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb121351d8b4e98a80d66f2448f5297"
I20260619 02:50:30.608749 32680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d478f7a7ca947c7929a6f19cd06ec5a" candidate_uuid: "d9c3c29089044903aa11c345b15b44af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e374899e2bd64b98a90e3aef2e23a951"
I20260619 02:50:30.608793 32546 raft_consensus.cc:3037] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:30.608830 32680 raft_consensus.cc:3037] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:50:30.609537 32546 raft_consensus.cc:2454] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9c3c29089044903aa11c345b15b44af in term 1.
I20260619 02:50:30.609637 32680 raft_consensus.cc:2454] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9c3c29089044903aa11c345b15b44af in term 1.
I20260619 02:50:30.609707 32750 leader_election.cc:302] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb121351d8b4e98a80d66f2448f5297, d9c3c29089044903aa11c345b15b44af; no voters:
I20260619 02:50:30.609824 414 raft_consensus.cc:2781] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:50:30.609970 414 raft_consensus.cc:686] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 1 LEADER]: Becoming Leader. State: Replica: d9c3c29089044903aa11c345b15b44af, State: Running, Role: LEADER
I20260619 02:50:30.610060 414 consensus_queue.cc:237] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb121351d8b4e98a80d66f2448f5297" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38113 } } peers { permanent_uuid: "d9c3c29089044903aa11c345b15b44af" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 44277 } } peers { permanent_uuid: "e374899e2bd64b98a90e3aef2e23a951" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 42137 } }
I20260619 02:50:30.610920 32400 catalog_manager.cc:5495] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af reported cstate change: term changed from 0 to 1, leader changed from <none> to d9c3c29089044903aa11c345b15b44af (127.21.164.195). New cstate: current_term: 1 leader_uuid: "d9c3c29089044903aa11c345b15b44af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb121351d8b4e98a80d66f2448f5297" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9c3c29089044903aa11c345b15b44af" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 44277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e374899e2bd64b98a90e3aef2e23a951" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 42137 } health_report { overall_health: UNKNOWN } } }
I20260619 02:50:30.619479 403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.617297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.21.164.195:44277)
0619 02:50:30.617338 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 02:50:30.617340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:50:30.617362 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:50:30.617439 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:50:30.617440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:50:30.617459 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:50:30.617513 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:50:30.617636 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.618091 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.618842 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:50:30.619028 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:50:30.619086 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:50:30.619228 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:50:30.619323 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:50:30.619373 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:50:30.619387 (+ 14us) server_negotiation.cc:299] Negotiation successful
0619 02:50:30.619403 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:50:30.621243 32680 raft_consensus.cc:1261] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951 [term 1 FOLLOWER]: Refusing update from remote peer d9c3c29089044903aa11c345b15b44af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:30.621294 32546 raft_consensus.cc:1261] T 7d478f7a7ca947c7929a6f19cd06ec5a P 8cb121351d8b4e98a80d66f2448f5297 [term 1 FOLLOWER]: Refusing update from remote peer d9c3c29089044903aa11c345b15b44af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:50:30.621598 416 consensus_queue.cc:1035] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb121351d8b4e98a80d66f2448f5297" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:30.621722 414 consensus_queue.cc:1035] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [LEADER]: Connected to new peer: Peer: permanent_uuid: "e374899e2bd64b98a90e3aef2e23a951" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 42137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:50:30.623962 419 mvcc.cc:205] Tried to move back new op lower bound from 7298406115822145536 to 7298406115779493888. Current Snapshot: MvccSnapshot[applied={T|T < 7298406115822145536}]
I20260619 02:50:30.624199 421 mvcc.cc:205] Tried to move back new op lower bound from 7298406115822145536 to 7298406115779493888. Current Snapshot: MvccSnapshot[applied={T|T < 7298406115822145536}]
I20260619 02:50:30.624198 420 mvcc.cc:205] Tried to move back new op lower bound from 7298406115822145536 to 7298406115779493888. Current Snapshot: MvccSnapshot[applied={T|T < 7298406115822145536}]
I20260619 02:50:30.626226 32400 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:30.626308 32400 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:50:30.627027 32400 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c4e0fe487fa4720980b20e0d2c24dce: Sending DeleteTablet for 3 replicas of tablet 7d478f7a7ca947c7929a6f19cd06ec5a
I20260619 02:50:30.627337 32660 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d478f7a7ca947c7929a6f19cd06ec5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:30 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:37122
I20260619 02:50:30.627377 32526 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d478f7a7ca947c7929a6f19cd06ec5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:30 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:47500
I20260619 02:50:30.627568 429 tablet_replica.cc:330] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: stopping tablet replica
I20260619 02:50:30.627643 326 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d478f7a7ca947c7929a6f19cd06ec5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:50:30 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:49008
I20260619 02:50:30.627674 429 raft_consensus.cc:2229] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:50:30.627767 429 raft_consensus.cc:2258] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:30.627815 431 tablet_replica.cc:330] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af: stopping tablet replica
I20260619 02:50:30.627898 431 raft_consensus.cc:2229] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:50:30.627969 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 32463
I20260619 02:50:30.627980 431 raft_consensus.cc:2258] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:50:30.628459 429 ts_tablet_manager.cc:1905] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:30.630218 429 ts_tablet_manager.cc:1918] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:30.630296 429 log.cc:1196] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/7d478f7a7ca947c7929a6f19cd06ec5a
I20260619 02:50:30.630383 431 ts_tablet_manager.cc:1905] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:50:30.630578 429 ts_tablet_manager.cc:1939] T 7d478f7a7ca947c7929a6f19cd06ec5a P e374899e2bd64b98a90e3aef2e23a951: Deleting consensus metadata
I20260619 02:50:30.630970 32387 catalog_manager.cc:4841] TS e374899e2bd64b98a90e3aef2e23a951 (127.21.164.194:42137): tablet 7d478f7a7ca947c7929a6f19cd06ec5a (table test-table [id=472e65a660cc4f3394e57b7ada217bbc]) successfully deleted
I20260619 02:50:30.631873 431 ts_tablet_manager.cc:1918] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:50:30.631945 431 log.cc:1196] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/7d478f7a7ca947c7929a6f19cd06ec5a
I20260619 02:50:30.632177 431 ts_tablet_manager.cc:1939] T 7d478f7a7ca947c7929a6f19cd06ec5a P d9c3c29089044903aa11c345b15b44af: Deleting consensus metadata
I20260619 02:50:30.632788 32385 catalog_manager.cc:4841] TS d9c3c29089044903aa11c345b15b44af (127.21.164.195:44277): tablet 7d478f7a7ca947c7929a6f19cd06ec5a (table test-table [id=472e65a660cc4f3394e57b7ada217bbc]) successfully deleted
W20260619 02:50:30.633646 32388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:38113 (error 108)
I20260619 02:50:30.633846 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 32597
I20260619 02:50:30.634421 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:50:30.633983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:38113 (local address 127.0.0.1:47502)
0619 02:50:30.634345 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 02:50:30.634387 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:38113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
W20260619 02:50:30.634816 32388 catalog_manager.cc:4584] TS 8cb121351d8b4e98a80d66f2448f5297 (127.21.164.193:38113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d478f7a7ca947c7929a6f19cd06ec5a: Network error: Client connection negotiation failed: client connection to 127.21.164.193:38113: connect: Connection refused (error 111)
I20260619 02:50:30.638737 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 32731
I20260619 02:50:30.642838 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 32369
2026-06-19T02:50:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1814 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:50:30 dist-test-slave-jt0t krb5kdc[435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:50:30 dist-test-slave-jt0t krb5kdc[435](info): set up 2 sockets
Jun 19 02:50:30 dist-test-slave-jt0t krb5kdc[435](info): commencing operation
krb5kdc: starting...
W20260619 02:50:31.601189 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:50:31 dist-test-slave-jt0t krb5kdc[435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:50:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.164.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.164.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/rangeradmin_127.21.164.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/rangeradmin_127.21.164.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.164.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.164.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/rangerlookup_127.21.164.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/rangerlookup_127.21.164.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.164.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.164.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/HTTP_127.21.164.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/HTTP_127.21.164.212@KRBTEST.COM.keytab.
I20260619 02:50:31.755131 22163 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1oh3JA/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/postgres -l logfile start
2026-06-19 02:50:34.078 UTC [465] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:50:34.078 UTC [465] LOG: listening on IPv4 address "127.21.164.212", port 50001
2026-06-19 02:50:34.082 UTC [465] LOG: listening on Unix socket "/tmp/.s.PGSQL.50001"
2026-06-19 02:50:34.109 UTC [468] LOG: database system was shut down at 2026-06-19 02:50:32 UTC
2026-06-19 02:50:34.112 UTC [465] LOG: database system is ready to accept connections
I20260619 02:50:34.977244 22163 mini_postgres.cc:96] Postgres bound to 50001
2026-06-19 02:50:34.983 UTC [476] FATAL: database "slave" does not exist
127.21.164.212:50001 - accepting connections
I20260619 02:50:34.983603 22163 mini_ranger.cc:161] Starting Ranger...
I20260619 02:50:34.997812 22163 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:50:35.442058 22163 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:50:35.442130 22163 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-admin
2026-06-19 02:50:35,714 [I] DB FLAVOR :POSTGRES
2026-06-19 02:50:35,715 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:50:35,715 [I] Checking connection..
2026-06-19 02:50:35,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:50:35,968 [I] Checking connection passed.
2026-06-19 02:50:35,969 [I] --------- Verifying version history table ---------
2026-06-19 02:50:35,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:50:36,223 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:50:36,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:50:36,466 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:50:36,467 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:50:36,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:50:36.722 UTC [540] WARNING: there is no transaction in progress
2026-06-19 02:50:36,727 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:50:36,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:50:36,983 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:50:36,983 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:50:36,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:50:37,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:50:37,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jt0t.c.gcp-upstream.internal','N') ;"
2026-06-19 02:50:37,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:50:37,965 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:50:37,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:50:38,215 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:50:38,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:50:38,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:50:38,682 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:50:38,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:50:39.615 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.632 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.648 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.664 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.679 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.695 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.749 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.756 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.770 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.781 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.791 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.805 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.814 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.820 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.827 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:39.834 UTC [648] WARNING: there is no transaction in progress
2026-06-19 02:50:40,459 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:50:40,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:50:40,705 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:50:40,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:50:40,956 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:50:40,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:50:41,211 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:50:41,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:50:41,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:50:41,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jt0t.c.gcp-upstream.internal';"
2026-06-19 02:50:41,962 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:50:41,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:50:42,213 [I] DB_PATCHES have already been applied
I20260619 02:50:42.218531 22163 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-admin:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1oh3JA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:50:42.218601 22163 mini_ranger.cc:193] Using host: 127.21.164.212
I20260619 02:50:42.221531 22163 mini_ranger.cc:227] Ranger admin URL: http://127.21.164.212:47403
Jun 19, 2026 2:50:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:50:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:50:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:50:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:50:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:50:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:50:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:50:43.203534 22163 mini_ranger.cc:160] Time spent starting Ranger: real 8.220s user 0.002s sys 0.007s
Jun 19, 2026 2:51:02 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:51:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 2:51:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:51:02 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:51:03 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:51:03.918326 22163 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.164.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.164.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/rangerkms_127.21.164.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/rangerkms_127.21.164.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.164.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.164.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.164.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/HTTP_127.21.164.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.164.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/HTTP_127.21.164.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:51:03 dist-test-slave-jt0t krb5kdc[435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837463, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:51:03.992794 22163 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:51:04.001231 22163 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:51:04.853214 22163 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:51:04.853297 22163 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-kms
I20260619 02:51:04.853314 22163 mini_ranger_kms.cc:227] Using postgres at 127.21.164.212:50001
2026-06-19 02:51:05,021 [I] DB FLAVOR :POSTGRES
2026-06-19 02:51:05,022 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:51:05,022 [I] Checking connection
2026-06-19 02:51:05,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:51:05,259 [I] connection success
2026-06-19 02:51:05,259 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:51:05,259 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:51:05,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:51:05,485 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:51:05,485 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:51:05,485 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:51:05,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:50001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:51:05,744 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:51:06.210335 22163 mini_ranger_kms.cc:310] Created kms service
I20260619 02:51:06.395639 22163 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:51:06.436897 22163 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:51:06 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837463, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:51:06.797173 22163 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:51:06.797294 22163 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-kms:/tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:51:06.797314 22163 mini_ranger_kms.cc:240] Using host: 127.21.164.212
I20260619 02:51:06.801157 22163 mini_ranger_kms.cc:276] Ranger KMS PID: 828
I20260619 02:51:06.801272 22163 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.164.212:51621
Jun 19, 2026 2:51:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:51:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:51:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:51:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:51:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:51:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 02:51:08.815790 22163 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.823s user 0.009s sys 0.142s
I20260619 02:51:08.815882 22163 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:51:08.815951 22163 mini_ranger_kms.cc:401] 127.21.164.212:51621/kms/v1/keys
Jun 19 02:51:08 dist-test-slave-jt0t krb5kdc[435](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.164.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 02:51:09 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.164.212@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
Jun 19 02:51:09 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.164.212@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
Jun 19 02:51:11 dist-test-slave-jt0t krb5kdc[435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:51:11.973356 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42779
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:42779
--ranger_config_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:51621/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:12.053952 879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:12.054189 879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:12.054212 879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:12.055408 879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:51:12.055450 879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:12.055464 879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:12.055474 879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:51:12.055485 879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:51:12.057121 879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.164.212:51621/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:42779
--ranger_config_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:51:12.057358 879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:12.057633 879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:12.063719 884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:12.063704 885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:12.063723 887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:12.064112 879 server_base.cc:1034] running on GCE node
I20260619 02:51:12.064329 879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:12.064683 879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:12.065824 879 hybrid_clock.cc:648] HybridClock initialized: now 1781837472065805 us; error 33 us; skew 500 ppm
Jun 19 02:51:12 dist-test-slave-jt0t krb5kdc[435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837472, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:12.068213 879 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:51:12.068691 879 webserver.cc:460] Webserver started at http://127.21.164.254:37255/ using document root <none> and password file <none>
I20260619 02:51:12.068907 879 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:12.068945 879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:12.069067 879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:12 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837472, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:51:12.132217 879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "6c92385088544ee883e8a2bf292ef6d6"
format_stamp: "Formatted at 2026-06-19 02:51:12 on dist-test-slave-jt0t"
server_key: "84af12f5cfd625b55986eebdc99dc25a"
server_key_iv: "6e3525cf51d977ce41e437dd86522563"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:12.132783 879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "6c92385088544ee883e8a2bf292ef6d6"
format_stamp: "Formatted at 2026-06-19 02:51:12 on dist-test-slave-jt0t"
server_key: "84af12f5cfd625b55986eebdc99dc25a"
server_key_iv: "6e3525cf51d977ce41e437dd86522563"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:12.134440 879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:51:12.149051 894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:12.149302 879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:51:12.149416 879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "6c92385088544ee883e8a2bf292ef6d6"
format_stamp: "Formatted at 2026-06-19 02:51:12 on dist-test-slave-jt0t"
server_key: "84af12f5cfd625b55986eebdc99dc25a"
server_key_iv: "6e3525cf51d977ce41e437dd86522563"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:12.149528 879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:12.171614 879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:12.173768 879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:12.173929 879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:12.179122 879 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:42779
I20260619 02:51:12.179159 946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:42779 every 8 connection(s)
I20260619 02:51:12.179553 879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:51:12.180254 947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:51:12.182541 947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6: Bootstrap starting.
I20260619 02:51:12.183125 947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6: Neither blocks nor log segments found. Creating new log.
I20260619 02:51:12.183396 947 log.cc:824] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:51:12.184126 947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6: No bootstrap required, opened a new log
I20260619 02:51:12.186206 947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c92385088544ee883e8a2bf292ef6d6" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42779 } }
I20260619 02:51:12.186358 947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:51:12.186380 947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c92385088544ee883e8a2bf292ef6d6, State: Initialized, Role: FOLLOWER
I20260619 02:51:12.186513 947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c92385088544ee883e8a2bf292ef6d6" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42779 } }
I20260619 02:51:12.186578 947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:51:12.186601 947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:51:12.186645 947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:51:12.187589 947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c92385088544ee883e8a2bf292ef6d6" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42779 } }
I20260619 02:51:12.187690 947 leader_election.cc:302] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c92385088544ee883e8a2bf292ef6d6; no voters:
I20260619 02:51:12.187856 947 leader_election.cc:288] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:51:12.187893 950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:51:12.188040 950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [term 1 LEADER]: Becoming Leader. State: Replica: 6c92385088544ee883e8a2bf292ef6d6, State: Running, Role: LEADER
I20260619 02:51:12.188113 947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:51:12.188190 950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c92385088544ee883e8a2bf292ef6d6" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42779 } }
I20260619 02:51:12.188629 952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c92385088544ee883e8a2bf292ef6d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c92385088544ee883e8a2bf292ef6d6" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42779 } } }
I20260619 02:51:12.188710 950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c92385088544ee883e8a2bf292ef6d6. Latest consensus state: current_term: 1 leader_uuid: "6c92385088544ee883e8a2bf292ef6d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c92385088544ee883e8a2bf292ef6d6" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42779 } } }
I20260619 02:51:12.188730 952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:51:12.189340 950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:51:12.189535 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 879
I20260619 02:51:12.189617 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:51:12.189751 22163 external_mini_cluster.cc:1428] Setting key ae8538dfe5fc0f9f73acc497e3b7e870
I20260619 02:51:12.190250 947 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:51:12.190598 964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:51:12.191067 964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:51:12.192497 964 catalog_manager.cc:1347] Generated new cluster ID: b2fe0e3b653e469ea84c9a1180ca58eb
I20260619 02:51:12.192549 964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:51:12 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:51:12.205233 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:12.190366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.21.164.254:42779)
0619 02:51:12.190531 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:12.190542 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:12.190574 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:51:12.190696 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:12.190703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:12.190761 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:12.190897 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:12.191933 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:12.195370 (+ 3437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:12.196466 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:12.197344 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:12.197454 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:12.200277 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:12.200282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:12.200282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:12.200293 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:12.203048 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:12.203474 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:12.203478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:12.203478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:12.203599 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:12.203794 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:12.203794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:12.203795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:12.204163 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:12.204323 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:12.204815 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 02:51:12.204944 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260619 02:51:12.205503 964 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:51:12.206771 964 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:51:12.228502 964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6: Generated new TSK 0
I20260619 02:51:12.228854 964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:51:13.010819 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:51621/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42779
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:13.097651 992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:13.097855 992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:13.097877 992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:13.099025 992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:13.099066 992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:13.099090 992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:51:13.100754 992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.164.212:51621/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:51:13.100986 992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:13.101224 992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:13.106102 998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:13.106161 997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:13.106115 1000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:13.106618 992 server_base.cc:1034] running on GCE node
I20260619 02:51:13.106832 992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:13.107125 992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:13.108309 992 hybrid_clock.cc:648] HybridClock initialized: now 1781837473108288 us; error 30 us; skew 500 ppm
Jun 19 02:51:13 dist-test-slave-jt0t krb5kdc[435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:13.111387 992 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:51:13.111982 992 webserver.cc:460] Webserver started at http://127.21.164.193:41923/ using document root <none> and password file <none>
I20260619 02:51:13.112262 992 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:13.112324 992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:13.112461 992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:13 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:51:13.124878 992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "d3d6a8c212a4423d8712a8399da8e216"
format_stamp: "Formatted at 2026-06-19 02:51:13 on dist-test-slave-jt0t"
server_key: "95e09184df653933d5379e9961cf6050"
server_key_iv: "b3dbc236e6ec0eb498baa6f1510df4be"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:13.125391 992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "d3d6a8c212a4423d8712a8399da8e216"
format_stamp: "Formatted at 2026-06-19 02:51:13 on dist-test-slave-jt0t"
server_key: "95e09184df653933d5379e9961cf6050"
server_key_iv: "b3dbc236e6ec0eb498baa6f1510df4be"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:13.127228 992 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:51:13.148795 1007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:13.148996 992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:51:13.149096 992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "d3d6a8c212a4423d8712a8399da8e216"
format_stamp: "Formatted at 2026-06-19 02:51:13 on dist-test-slave-jt0t"
server_key: "95e09184df653933d5379e9961cf6050"
server_key_iv: "b3dbc236e6ec0eb498baa6f1510df4be"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:13.149194 992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:13.164546 992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:13.166522 992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:13.166669 992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:13.166911 992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:51:13.167258 992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:51:13.167294 992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:13.167325 992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:51:13.167361 992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:13.175598 992 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:43945
I20260619 02:51:13.175688 1121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:43945 every 8 connection(s)
I20260619 02:51:13.176074 992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:51:13.176855 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 992
I20260619 02:51:13.176967 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:51:13.177160 22163 external_mini_cluster.cc:1428] Setting key bfcabbaef54f1319ff1db4b34be54a7a
Jun 19 02:51:13 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:51:13.189591 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.177976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:36275 (local address 127.21.164.254:42779)
0619 02:51:13.178723 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:13.178725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:13.178733 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:51:13.178758 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:13.178759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:13.178771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:13.181329 (+ 2558us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:13.181360 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.181839 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.183273 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.183564 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.183630 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.186275 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:13.186277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:13.186277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:13.186285 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:13.188030 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:13.188514 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:13.188516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:13.188516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:13.188581 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:13.188900 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:13.188901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:13.188902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:13.189044 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:13.189100 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:13.189400 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 02:51:13.189485 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":83,"threads_started":1}
I20260619 02:51:13.189730 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.177475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42779 (local address 127.21.164.193:36275)
0619 02:51:13.177866 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:13.177914 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:13.178520 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:13.180763 (+ 2243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:13.180769 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:13.181005 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:51:13.181251 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.181257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.181929 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.181930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.183171 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.183174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.183700 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.183701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.183864 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.184467 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:51:13.184477 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:51:13.186161 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:51:13.188127 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:13.188133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:13.188136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:13.188431 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:13.188676 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:13.188677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:13.188677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:13.188825 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:13.189124 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:51:13.189131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:51:13.189261 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 02:51:13.189348 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 02:51:13.189495 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":88,"threads_started":1}
I20260619 02:51:13.190280 1122 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42779
I20260619 02:51:13.190377 1122 heartbeater.cc:464] Registering TS with master...
I20260619 02:51:13.190562 1122 heartbeater.cc:511] Master 127.21.164.254:42779 requested a full tablet report, sending...
I20260619 02:51:13.191078 911 ts_manager.cc:194] Registered new tserver with Master: d3d6a8c212a4423d8712a8399da8e216 (127.21.164.193:43945)
I20260619 02:51:13.191702 911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:36275
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:51:13.224056 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:51621/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42779
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:13.331970 1129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:13.332247 1129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:13.332286 1129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:13.334081 1129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:13.334148 1129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:13.334188 1129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:51:13.336402 1129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.164.212:51621/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:51:13.336850 1129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:13.337152 1129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:13.343524 1139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:13.343544 1137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:13.343814 1136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:13.344053 1129 server_base.cc:1034] running on GCE node
I20260619 02:51:13.344281 1129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:13.344553 1129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:13.345716 1129 hybrid_clock.cc:648] HybridClock initialized: now 1781837473345702 us; error 27 us; skew 500 ppm
Jun 19 02:51:13 dist-test-slave-jt0t krb5kdc[435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:13.349014 1129 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:51:13.349560 1129 webserver.cc:460] Webserver started at http://127.21.164.194:41827/ using document root <none> and password file <none>
I20260619 02:51:13.349828 1129 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:13.349877 1129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:13.349998 1129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:13 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
Jun 19 02:51:13 dist-test-slave-jt0t krb5kdc[435](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:13.374519 1129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "92aa6ff689e04ce1a1148fe9be20b967"
format_stamp: "Formatted at 2026-06-19 02:51:13 on dist-test-slave-jt0t"
server_key: "224b39c725ecaf1d12ce0421495cd095"
server_key_iv: "cc79fff4a9ae25acc1d6081874c1f6a9"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:13.374970 1129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "92aa6ff689e04ce1a1148fe9be20b967"
format_stamp: "Formatted at 2026-06-19 02:51:13 on dist-test-slave-jt0t"
server_key: "224b39c725ecaf1d12ce0421495cd095"
server_key_iv: "cc79fff4a9ae25acc1d6081874c1f6a9"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:13.377671 1129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:51:13.412173 1146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:13.412463 1129 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260619 02:51:13.412551 1129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "92aa6ff689e04ce1a1148fe9be20b967"
format_stamp: "Formatted at 2026-06-19 02:51:13 on dist-test-slave-jt0t"
server_key: "224b39c725ecaf1d12ce0421495cd095"
server_key_iv: "cc79fff4a9ae25acc1d6081874c1f6a9"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:13.412760 1129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:13.464820 1129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:13.467267 1129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:13.467471 1129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:13.467720 1129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:51:13.468097 1129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:51:13.468171 1129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:13.468237 1129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:51:13.468303 1129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:13.476697 1129 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:41487
I20260619 02:51:13.477200 1129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:51:13.483626 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 1129
I20260619 02:51:13.483747 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:51:13.483937 22163 external_mini_cluster.cc:1428] Setting key 086113ed0fc6853738e42e0b6376fabf
I20260619 02:51:13.478091 1260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:41487 every 8 connection(s)
Jun 19 02:51:13 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:51:13.492352 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.478755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:35651 (local address 127.21.164.254:42779)
0619 02:51:13.478806 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:13.478808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:13.479044 (+ 236us) server_negotiation.cc:407] Connection header received
0619 02:51:13.479754 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:13.479756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:13.479775 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:13.479804 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:13.480702 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.481450 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.482982 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.483329 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.483379 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.488782 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:13.488784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:13.488785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:13.488793 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:13.490602 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:13.491116 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:13.491117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:13.491118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:13.491179 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:13.491495 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:13.491496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:13.491496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:13.491664 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:13.491711 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:13.492117 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 02:51:13.492220 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:51:13.493242 1263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.478740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42779 (local address 127.21.164.194:35651)
0619 02:51:13.478903 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:13.478954 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:13.479626 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:13.480008 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:13.480013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:13.480268 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:51:13.480563 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.480570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.481591 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.481592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.482880 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.482883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.484357 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.484358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.484676 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.486727 (+ 2051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:51:13.486740 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:51:13.488600 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:51:13.490704 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:13.490713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:13.490716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:13.491025 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:13.491252 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:13.491253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:13.491253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:13.491417 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:13.491761 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:51:13.491768 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:51:13.491947 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 02:51:13.492802 (+ 855us) client_negotiation.cc:240] Negotiation successful
0619 02:51:13.492988 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260619 02:51:13.494004 1261 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42779
I20260619 02:51:13.494112 1261 heartbeater.cc:464] Registering TS with master...
I20260619 02:51:13.494323 1261 heartbeater.cc:511] Master 127.21.164.254:42779 requested a full tablet report, sending...
I20260619 02:51:13.494755 911 ts_manager.cc:194] Registered new tserver with Master: 92aa6ff689e04ce1a1148fe9be20b967 (127.21.164.194:41487)
I20260619 02:51:13.495350 911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:35651
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:51:13.519811 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:51621/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42779
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:13.646456 1266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:13.646673 1266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:13.646697 1266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:13.647907 1266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:13.647953 1266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:13.647979 1266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:51:13.649550 1266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.164.212:51621/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:51:13.649777 1266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:13.650000 1266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:13.655862 1272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:13.655872 1271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:13.656080 1266 server_base.cc:1034] running on GCE node
W20260619 02:51:13.660593 1274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:13.660945 1266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:13.661238 1266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:13.662420 1266 hybrid_clock.cc:648] HybridClock initialized: now 1781837473662396 us; error 42 us; skew 500 ppm
Jun 19 02:51:13 dist-test-slave-jt0t krb5kdc[435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:13.665087 1266 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:51:13.665620 1266 webserver.cc:460] Webserver started at http://127.21.164.195:42295/ using document root <none> and password file <none>
I20260619 02:51:13.665843 1266 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:13.665894 1266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:13.666028 1266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:13 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:51:13.680894 1266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "0f532f91b0a141b3a82b8272f4e19d21"
format_stamp: "Formatted at 2026-06-19 02:51:13 on dist-test-slave-jt0t"
server_key: "e458498e10ebcdd8ce57bc8217ea8448"
server_key_iv: "20ceee8524fdc5c7d005ed2209b69047"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:13.681452 1266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "0f532f91b0a141b3a82b8272f4e19d21"
format_stamp: "Formatted at 2026-06-19 02:51:13 on dist-test-slave-jt0t"
server_key: "e458498e10ebcdd8ce57bc8217ea8448"
server_key_iv: "20ceee8524fdc5c7d005ed2209b69047"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:13.694751 1266 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.001s sys 0.000s
I20260619 02:51:13.724797 1281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:13.725029 1266 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.000s sys 0.001s
I20260619 02:51:13.725147 1266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "0f532f91b0a141b3a82b8272f4e19d21"
format_stamp: "Formatted at 2026-06-19 02:51:13 on dist-test-slave-jt0t"
server_key: "e458498e10ebcdd8ce57bc8217ea8448"
server_key_iv: "20ceee8524fdc5c7d005ed2209b69047"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:13.725274 1266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:13.771692 1266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:13.774159 1266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:13.774375 1266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:13.774650 1266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:51:13.775029 1266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:51:13.775115 1266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:13.775184 1266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:51:13.775234 1266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:13.782564 1266 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:38915
I20260619 02:51:13.783059 1266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:51:13.784580 1394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:38915 every 8 connection(s)
I20260619 02:51:13.791252 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 1266
I20260619 02:51:13.791368 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:51:13.791538 22163 external_mini_cluster.cc:1428] Setting key ce7263a43ac1e7f2e47d96a83dc0ae62
Jun 19 02:51:13 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:51:13.802763 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.785263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:48245 (local address 127.21.164.254:42779)
0619 02:51:13.785308 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:13.785310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:13.792951 (+ 7641us) server_negotiation.cc:407] Connection header received
0619 02:51:13.793612 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:13.793626 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:13.793638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:13.793667 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:13.794514 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.795311 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.796828 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.797128 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.797168 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.799703 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:13.799705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:13.799705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:13.799712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:13.801256 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:13.801721 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:13.801723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:13.801723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:13.801778 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:13.802030 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:13.802031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:13.802031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:13.802158 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:13.802199 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:13.802572 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 02:51:13.802663 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:51:13.803747 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.785276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42779 (local address 127.21.164.195:48245)
0619 02:51:13.792800 (+ 7524us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:13.792867 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:13.793501 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:13.793874 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:13.793879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:13.794129 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:51:13.794415 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.794420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.795423 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.795424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.796740 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.796743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.797252 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.797253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.797405 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.798001 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:51:13.798012 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:51:13.799587 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:51:13.801347 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:13.801353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:13.801356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:13.801643 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:13.801839 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:13.801840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:13.801840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:13.801968 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:13.802240 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:51:13.802246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:51:13.802415 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 02:51:13.803320 (+ 905us) client_negotiation.cc:240] Negotiation successful
0619 02:51:13.803492 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7465,"thread_start_us":84,"threads_started":1}
I20260619 02:51:13.804586 1395 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42779
I20260619 02:51:13.804734 1395 heartbeater.cc:464] Registering TS with master...
I20260619 02:51:13.804924 1395 heartbeater.cc:511] Master 127.21.164.254:42779 requested a full tablet report, sending...
I20260619 02:51:13.805321 911 ts_manager.cc:194] Registered new tserver with Master: 0f532f91b0a141b3a82b8272f4e19d21 (127.21.164.195:38915)
I20260619 02:51:13.805823 911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:48245
I20260619 02:51:13.815132 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:51:13.824529 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.816885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.21.164.254:42779)
0619 02:51:13.816939 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:13.816941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:13.817146 (+ 205us) server_negotiation.cc:407] Connection header received
0619 02:51:13.817242 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:13.817243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:13.817253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:13.817281 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:13.818017 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.818801 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.819876 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.820171 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.820210 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.821766 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:13.821768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:13.821768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:13.821775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:13.823309 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:13.823730 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:13.823731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:13.823732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:13.823786 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:13.823964 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:13.823965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:13.823965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:13.824084 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:13.824152 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:13.824355 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 02:51:13.824430 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:51:13.826772 911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:51:13.828938 911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:51:13.838603 1409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.832122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:38915 (local address 127.0.0.1:58212)
0619 02:51:13.833646 (+ 1524us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:13.833653 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:13.833686 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:13.833937 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:13.833940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:13.833942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:51:13.834013 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.834015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.835581 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.835582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.837209 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.837212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.838299 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.838300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.838378 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.838393 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 02:51:13.838458 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:51:13.838511 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1514,"thread_start_us":58,"threads_started":1}
I20260619 02:51:13.839182 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.832314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.21.164.195:38915)
0619 02:51:13.833276 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:13.833281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:13.833730 (+ 449us) server_negotiation.cc:407] Connection header received
0619 02:51:13.833773 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:13.833775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:13.833791 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:13.833852 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:51:13.834079 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.835489 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.837310 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.838132 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.838211 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.838267 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:13.839060 (+ 793us) server_negotiation.cc:299] Negotiation successful
0619 02:51:13.839101 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":65,"threads_started":1}
I20260619 02:51:13.840030 1329 tablet_service.cc:1467] Processing CreateTablet for tablet edb2504468234f93b210f421197bdc2a (DEFAULT_TABLE table=test-table [id=901a72fa7d384233b9787464c4c9a66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:51:13.840381 1329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edb2504468234f93b210f421197bdc2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:51:13.844578 1414 tablet_bootstrap.cc:492] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21: Bootstrap starting.
I20260619 02:51:13.845660 1414 tablet_bootstrap.cc:654] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21: Neither blocks nor log segments found. Creating new log.
I20260619 02:51:13.846029 1414 log.cc:824] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21: Log is configured to *not* fsync() on all Append() calls
I20260619 02:51:13.846784 1414 tablet_bootstrap.cc:492] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21: No bootstrap required, opened a new log
I20260619 02:51:13.846844 1414 ts_tablet_manager.cc:1397] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 02:51:13.848680 1414 raft_consensus.cc:348] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d6a8c212a4423d8712a8399da8e216" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 43945 } } peers { permanent_uuid: "92aa6ff689e04ce1a1148fe9be20b967" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41487 } } peers { permanent_uuid: "0f532f91b0a141b3a82b8272f4e19d21" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 38915 } }
I20260619 02:51:13.848842 1414 raft_consensus.cc:374] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:51:13.848869 1414 raft_consensus.cc:729] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f532f91b0a141b3a82b8272f4e19d21, State: Initialized, Role: FOLLOWER
I20260619 02:51:13.848977 1414 consensus_queue.cc:260] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d6a8c212a4423d8712a8399da8e216" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 43945 } } peers { permanent_uuid: "92aa6ff689e04ce1a1148fe9be20b967" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41487 } } peers { permanent_uuid: "0f532f91b0a141b3a82b8272f4e19d21" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 38915 } }
I20260619 02:51:13.849215 1414 ts_tablet_manager.cc:1428] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:51:13.849436 1395 heartbeater.cc:503] Master 127.21.164.254:42779 was elected leader, sending a full tablet report...
I20260619 02:51:13.855396 1410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.831776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:43945 (local address 127.0.0.1:33348)
0619 02:51:13.833345 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:13.833390 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:13.833453 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:13.844970 (+ 11517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:13.844972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:13.844975 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:51:13.845036 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.845038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.846893 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.846894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.847833 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.847836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.855101 (+ 7265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.855102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.855190 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.855196 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:51:13.855250 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:51:13.855301 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1540,"thread_start_us":74,"threads_started":1}
I20260619 02:51:13.859135 1408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.831992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:41487 (local address 127.0.0.1:39908)
0619 02:51:13.833543 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:13.833550 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:13.833579 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:13.855194 (+ 21615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:13.855195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:13.855199 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:51:13.855250 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.855252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.856802 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.856803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.857958 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.857960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.858906 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.858907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.858974 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.858979 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:51:13.859006 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 02:51:13.859045 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1540,"thread_start_us":51,"threads_started":1}
I20260619 02:51:13.859664 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.833167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.21.164.194:41487)
0619 02:51:13.854165 (+ 20998us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:13.854171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:13.854210 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:51:13.854250 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:13.854252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:13.854270 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:13.855321 (+ 1051us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:51:13.855397 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.856709 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.858071 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.858812 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.859433 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.859508 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:13.859536 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 02:51:13.859576 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20953,"thread_start_us":81,"threads_started":1}
I20260619 02:51:13.860611 1195 tablet_service.cc:1467] Processing CreateTablet for tablet edb2504468234f93b210f421197bdc2a (DEFAULT_TABLE table=test-table [id=901a72fa7d384233b9787464c4c9a66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:51:13.860867 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.833029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.21.164.193:43945)
0619 02:51:13.844755 (+ 11726us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:13.844771 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:13.844784 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:51:13.844823 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:13.844825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:13.844843 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:13.845114 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:51:13.845182 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.846783 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.853784 (+ 7001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.855002 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.855076 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.855133 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:13.860725 (+ 5592us) server_negotiation.cc:299] Negotiation successful
0619 02:51:13.860793 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11685,"thread_start_us":55,"threads_started":1}
I20260619 02:51:13.861032 1195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edb2504468234f93b210f421197bdc2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:51:13.861578 1037 tablet_service.cc:1467] Processing CreateTablet for tablet edb2504468234f93b210f421197bdc2a (DEFAULT_TABLE table=test-table [id=901a72fa7d384233b9787464c4c9a66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:51:13.861915 1037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edb2504468234f93b210f421197bdc2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:51:13.868854 1417 tablet_bootstrap.cc:492] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216: Bootstrap starting.
I20260619 02:51:13.869627 1417 tablet_bootstrap.cc:654] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216: Neither blocks nor log segments found. Creating new log.
I20260619 02:51:13.869957 1417 log.cc:824] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216: Log is configured to *not* fsync() on all Append() calls
I20260619 02:51:13.870354 1418 tablet_bootstrap.cc:492] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: Bootstrap starting.
I20260619 02:51:13.871420 1418 tablet_bootstrap.cc:654] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: Neither blocks nor log segments found. Creating new log.
I20260619 02:51:13.871770 1418 log.cc:824] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: Log is configured to *not* fsync() on all Append() calls
I20260619 02:51:13.881525 1417 tablet_bootstrap.cc:492] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216: No bootstrap required, opened a new log
I20260619 02:51:13.881639 1417 ts_tablet_manager.cc:1397] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
Jun 19 02:51:13 dist-test-slave-jt0t krb5kdc[435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837473, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:51:13.884233 1417 raft_consensus.cc:348] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d6a8c212a4423d8712a8399da8e216" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 43945 } } peers { permanent_uuid: "92aa6ff689e04ce1a1148fe9be20b967" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41487 } } peers { permanent_uuid: "0f532f91b0a141b3a82b8272f4e19d21" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 38915 } }
I20260619 02:51:13.884439 1417 raft_consensus.cc:374] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:51:13.884478 1417 raft_consensus.cc:729] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d6a8c212a4423d8712a8399da8e216, State: Initialized, Role: FOLLOWER
I20260619 02:51:13.884684 1417 consensus_queue.cc:260] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d6a8c212a4423d8712a8399da8e216" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 43945 } } peers { permanent_uuid: "92aa6ff689e04ce1a1148fe9be20b967" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41487 } } peers { permanent_uuid: "0f532f91b0a141b3a82b8272f4e19d21" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 38915 } }
I20260619 02:51:13.885015 1417 ts_tablet_manager.cc:1428] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 02:51:13.885142 1122 heartbeater.cc:503] Master 127.21.164.254:42779 was elected leader, sending a full tablet report...
I20260619 02:51:13.886570 1418 tablet_bootstrap.cc:492] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: No bootstrap required, opened a new log
I20260619 02:51:13.886654 1418 ts_tablet_manager.cc:1397] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260619 02:51:13.888751 1418 raft_consensus.cc:348] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d6a8c212a4423d8712a8399da8e216" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 43945 } } peers { permanent_uuid: "92aa6ff689e04ce1a1148fe9be20b967" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41487 } } peers { permanent_uuid: "0f532f91b0a141b3a82b8272f4e19d21" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 38915 } }
I20260619 02:51:13.888948 1418 raft_consensus.cc:374] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:51:13.888983 1418 raft_consensus.cc:729] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92aa6ff689e04ce1a1148fe9be20b967, State: Initialized, Role: FOLLOWER
I20260619 02:51:13.889127 1418 consensus_queue.cc:260] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d6a8c212a4423d8712a8399da8e216" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 43945 } } peers { permanent_uuid: "92aa6ff689e04ce1a1148fe9be20b967" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41487 } } peers { permanent_uuid: "0f532f91b0a141b3a82b8272f4e19d21" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 38915 } }
I20260619 02:51:13.889410 1418 ts_tablet_manager.cc:1428] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:51:13.889551 1261 heartbeater.cc:503] Master 127.21.164.254:42779 was elected leader, sending a full tablet report...
I20260619 02:51:13.919726 1416 raft_consensus.cc:482] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:51:13.919869 1416 raft_consensus.cc:504] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d6a8c212a4423d8712a8399da8e216" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 43945 } } peers { permanent_uuid: "92aa6ff689e04ce1a1148fe9be20b967" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41487 } } peers { permanent_uuid: "0f532f91b0a141b3a82b8272f4e19d21" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 38915 } }
I20260619 02:51:13.920265 1416 leader_election.cc:288] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3d6a8c212a4423d8712a8399da8e216 (127.21.164.193:43945), 92aa6ff689e04ce1a1148fe9be20b967 (127.21.164.194:41487)
I20260619 02:51:13.924587 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.920490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:43945 (local address 127.21.164.195:46801)
0619 02:51:13.920746 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:13.920757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:13.920800 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:13.921199 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:13.921200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:13.921202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:51:13.921397 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.921400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.922308 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.922310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.923526 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.923529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.924330 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.924331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.924416 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.924420 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:51:13.924462 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:51:13.924504 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260619 02:51:13.924819 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.920940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:46801 (local address 127.21.164.193:43945)
0619 02:51:13.921024 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:13.921025 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:13.921033 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:51:13.921059 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:13.921060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:13.921070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:13.921114 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:51:13.921473 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.922219 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.923635 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.924220 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.924287 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.924298 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:13.924713 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 02:51:13.924757 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:51:13.925426 1074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2504468234f93b210f421197bdc2a" candidate_uuid: "0f532f91b0a141b3a82b8272f4e19d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d6a8c212a4423d8712a8399da8e216" is_pre_election: true
I20260619 02:51:13.925598 1074 raft_consensus.cc:2454] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f532f91b0a141b3a82b8272f4e19d21 in term 0.
I20260619 02:51:13.925851 1284 leader_election.cc:302] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f532f91b0a141b3a82b8272f4e19d21, d3d6a8c212a4423d8712a8399da8e216; no voters:
I20260619 02:51:13.925977 1416 raft_consensus.cc:2781] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:51:13.926015 1416 raft_consensus.cc:482] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:51:13.926034 1416 raft_consensus.cc:3037] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:51:13.926928 1416 raft_consensus.cc:504] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d6a8c212a4423d8712a8399da8e216" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 43945 } } peers { permanent_uuid: "92aa6ff689e04ce1a1148fe9be20b967" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41487 } } peers { permanent_uuid: "0f532f91b0a141b3a82b8272f4e19d21" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 38915 } }
I20260619 02:51:13.927078 1416 leader_election.cc:288] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [CANDIDATE]: Term 1 election: Requested vote from peers d3d6a8c212a4423d8712a8399da8e216 (127.21.164.193:43945), 92aa6ff689e04ce1a1148fe9be20b967 (127.21.164.194:41487)
I20260619 02:51:13.927255 1074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2504468234f93b210f421197bdc2a" candidate_uuid: "0f532f91b0a141b3a82b8272f4e19d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d6a8c212a4423d8712a8399da8e216"
I20260619 02:51:13.927338 1074 raft_consensus.cc:3037] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:51:13.928133 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.920603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:41487 (local address 127.21.164.195:47131)
0619 02:51:13.924958 (+ 4355us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:13.924968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:13.925013 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:13.925316 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:13.925317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:13.925319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:51:13.925350 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.925351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.926154 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.926155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.927102 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:13.927104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.927927 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.927928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:13.927983 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.927986 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:51:13.928028 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:51:13.928063 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4344,"thread_start_us":735,"threads_started":1}
I20260619 02:51:13.928227 1074 raft_consensus.cc:2454] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f532f91b0a141b3a82b8272f4e19d21 in term 1.
I20260619 02:51:13.928385 1284 leader_election.cc:302] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f532f91b0a141b3a82b8272f4e19d21, d3d6a8c212a4423d8712a8399da8e216; no voters:
I20260619 02:51:13.928397 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.920985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:47131 (local address 127.21.164.194:41487)
0619 02:51:13.921167 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:13.921169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:13.925061 (+ 3892us) server_negotiation.cc:407] Connection header received
0619 02:51:13.925092 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:13.925093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:13.925102 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:13.925142 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:51:13.925415 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.926050 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.927318 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.927819 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.927887 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.927896 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:13.928294 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 02:51:13.928338 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260619 02:51:13.928455 1416 raft_consensus.cc:2781] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:51:13.928602 1416 raft_consensus.cc:686] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 1 LEADER]: Becoming Leader. State: Replica: 0f532f91b0a141b3a82b8272f4e19d21, State: Running, Role: LEADER
I20260619 02:51:13.928723 1213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2504468234f93b210f421197bdc2a" candidate_uuid: "0f532f91b0a141b3a82b8272f4e19d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92aa6ff689e04ce1a1148fe9be20b967" is_pre_election: true
I20260619 02:51:13.928738 1416 consensus_queue.cc:237] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d6a8c212a4423d8712a8399da8e216" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 43945 } } peers { permanent_uuid: "92aa6ff689e04ce1a1148fe9be20b967" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41487 } } peers { permanent_uuid: "0f532f91b0a141b3a82b8272f4e19d21" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 38915 } }
I20260619 02:51:13.928856 1213 raft_consensus.cc:2454] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f532f91b0a141b3a82b8272f4e19d21 in term 0.
I20260619 02:51:13.928962 1215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb2504468234f93b210f421197bdc2a" candidate_uuid: "0f532f91b0a141b3a82b8272f4e19d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92aa6ff689e04ce1a1148fe9be20b967"
I20260619 02:51:13.929009 1215 raft_consensus.cc:3037] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:51:13.929579 910 catalog_manager.cc:5495] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f532f91b0a141b3a82b8272f4e19d21 (127.21.164.195). New cstate: current_term: 1 leader_uuid: "0f532f91b0a141b3a82b8272f4e19d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d6a8c212a4423d8712a8399da8e216" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 43945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92aa6ff689e04ce1a1148fe9be20b967" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f532f91b0a141b3a82b8272f4e19d21" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 38915 } health_report { overall_health: HEALTHY } } }
I20260619 02:51:13.929863 1215 raft_consensus.cc:2454] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f532f91b0a141b3a82b8272f4e19d21 in term 1.
W20260619 02:51:13.934940 1123 tablet.cc:2369] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:51:13.967314 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:13.963904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.21.164.195:38915)
0619 02:51:13.963951 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:13.963953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:13.963965 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:51:13.964063 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:13.964064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:13.964090 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:13.964152 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:51:13.964343 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.965093 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.966181 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:13.966469 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:13.966522 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:13.966723 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:51:13.966858 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:51:13.966926 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:13.967191 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 02:51:13.967212 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:51:13.975301 1215 raft_consensus.cc:1261] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967 [term 1 FOLLOWER]: Refusing update from remote peer 0f532f91b0a141b3a82b8272f4e19d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:51:13.975755 1424 consensus_queue.cc:1035] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92aa6ff689e04ce1a1148fe9be20b967" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:51:13.978035 1074 raft_consensus.cc:1261] T edb2504468234f93b210f421197bdc2a P d3d6a8c212a4423d8712a8399da8e216 [term 1 FOLLOWER]: Refusing update from remote peer 0f532f91b0a141b3a82b8272f4e19d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:51:13.978364 1424 consensus_queue.cc:1035] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d6a8c212a4423d8712a8399da8e216" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:51:13.978874 1262 tablet.cc:2369] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:51:13.987425 1430 mvcc.cc:205] Tried to move back new op lower bound from 7298406293399318528 to 7298406293212704768. Current Snapshot: MvccSnapshot[applied={T|T < 7298406293399318528}]
I20260619 02:51:13.988634 1428 mvcc.cc:205] Tried to move back new op lower bound from 7298406293399318528 to 7298406293212704768. Current Snapshot: MvccSnapshot[applied={T|T < 7298406293399318528}]
I20260619 02:51:13.991355 910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:51:13.991453 910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:51:13.992295 910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c92385088544ee883e8a2bf292ef6d6: Sending DeleteTablet for 3 replicas of tablet edb2504468234f93b210f421197bdc2a
I20260619 02:51:13.992766 1037 tablet_service.cc:1514] Processing DeleteTablet for tablet edb2504468234f93b210f421197bdc2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:51:13 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:33348
I20260619 02:51:13.993491 1195 tablet_service.cc:1514] Processing DeleteTablet for tablet edb2504468234f93b210f421197bdc2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:51:13 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:39908
I20260619 02:51:13.993978 1329 tablet_service.cc:1514] Processing DeleteTablet for tablet edb2504468234f93b210f421197bdc2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:51:13 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:58212
I20260619 02:51:13.994624 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 992
I20260619 02:51:14.004165 1437 tablet_replica.cc:330] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: stopping tablet replica
I20260619 02:51:14.004369 1438 tablet_replica.cc:330] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21: stopping tablet replica
I20260619 02:51:14.004776 1437 raft_consensus.cc:2229] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:51:14.005215 1438 raft_consensus.cc:2229] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:51:14.005443 1438 raft_consensus.cc:2258] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:51:14.005568 1437 raft_consensus.cc:2258] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:51:14.013151 1437 ts_tablet_manager.cc:1905] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:51:14.016026 1437 ts_tablet_manager.cc:1918] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:51:14.016193 1437 log.cc:1196] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/edb2504468234f93b210f421197bdc2a
I20260619 02:51:14.016546 1437 ts_tablet_manager.cc:1939] T edb2504468234f93b210f421197bdc2a P 92aa6ff689e04ce1a1148fe9be20b967: Deleting consensus metadata
I20260619 02:51:14.016979 1438 ts_tablet_manager.cc:1905] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:51:14.017530 895 catalog_manager.cc:4841] TS 92aa6ff689e04ce1a1148fe9be20b967 (127.21.164.194:41487): tablet edb2504468234f93b210f421197bdc2a (table test-table [id=901a72fa7d384233b9787464c4c9a66c]) successfully deleted
W20260619 02:51:14.017803 897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:43945 (error 108)
I20260619 02:51:14.020781 1438 ts_tablet_manager.cc:1918] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:51:14.020833 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 1129
I20260619 02:51:14.020870 1438 log.cc:1196] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/edb2504468234f93b210f421197bdc2a
I20260619 02:51:14.021109 1438 ts_tablet_manager.cc:1939] T edb2504468234f93b210f421197bdc2a P 0f532f91b0a141b3a82b8272f4e19d21: Deleting consensus metadata
I20260619 02:51:14.021831 898 catalog_manager.cc:4841] TS 0f532f91b0a141b3a82b8272f4e19d21 (127.21.164.195:38915): tablet edb2504468234f93b210f421197bdc2a (table test-table [id=901a72fa7d384233b9787464c4c9a66c]) successfully deleted
I20260619 02:51:14.029323 1408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:14.029169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:43945 (local address 127.0.0.1:33358)
0619 02:51:14.029228 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:14.029275 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:43945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 02:51:14.029603 897 catalog_manager.cc:4584] TS d3d6a8c212a4423d8712a8399da8e216 (127.21.164.193:43945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edb2504468234f93b210f421197bdc2a: Network error: Client connection negotiation failed: client connection to 127.21.164.193:43945: connect: Connection refused (error 111)
I20260619 02:51:14.030386 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 1266
I20260619 02:51:14.037492 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 879
I20260619 02:51:14.043021 22163 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:51:14.377455 22163 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:51:14.377556 22163 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:51:14.712018 22163 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:51:14.712 UTC [465] LOG: received smart shutdown request
2026-06-19 02:51:14.715 UTC [465] LOG: background worker "logical replication launcher" (PID 474) exited with exit code 1
2026-06-19 02:51:14.715 UTC [469] LOG: shutting down
2026-06-19 02:51:14.880 UTC [465] LOG: database system is shut down
2026-06-19T02:51:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44319 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:51:15 dist-test-slave-jt0t krb5kdc[1448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:51:15 dist-test-slave-jt0t krb5kdc[1448](info): set up 2 sockets
Jun 19 02:51:15 dist-test-slave-jt0t krb5kdc[1448](info): commencing operation
krb5kdc: starting...
W20260619 02:51:15.916359 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:51:15 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:51:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:51:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.164.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.164.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/rangeradmin_127.21.164.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/rangeradmin_127.21.164.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.164.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.164.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/rangerlookup_127.21.164.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/rangerlookup_127.21.164.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.164.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.164.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/HTTP_127.21.164.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/HTTP_127.21.164.212@KRBTEST.COM.keytab.
I20260619 02:51:16.070998 22163 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1oh3JA/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/postgres -l logfile start
2026-06-19 02:51:18.430 UTC [1478] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:51:18.430 UTC [1478] LOG: listening on IPv4 address "127.21.164.212", port 38943
2026-06-19 02:51:18.433 UTC [1478] LOG: listening on Unix socket "/tmp/.s.PGSQL.38943"
2026-06-19 02:51:18.461 UTC [1481] LOG: database system was shut down at 2026-06-19 02:51:17 UTC
2026-06-19 02:51:18.464 UTC [1478] LOG: database system is ready to accept connections
I20260619 02:51:19.328248 22163 mini_postgres.cc:96] Postgres bound to 38943
2026-06-19 02:51:19.333 UTC [1489] FATAL: database "slave" does not exist
127.21.164.212:38943 - accepting connections
I20260619 02:51:19.334440 22163 mini_ranger.cc:161] Starting Ranger...
I20260619 02:51:19.348384 22163 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:51:19.788431 22163 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:51:19.788513 22163 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-admin
2026-06-19 02:51:20,066 [I] DB FLAVOR :POSTGRES
2026-06-19 02:51:20,067 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:51:20,067 [I] Checking connection..
2026-06-19 02:51:20,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:51:20,312 [I] Checking connection passed.
2026-06-19 02:51:20,312 [I] --------- Verifying version history table ---------
2026-06-19 02:51:20,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:51:20,545 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:51:20,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:51:20,799 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:51:20,799 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:51:20,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:51:21.029 UTC [1553] WARNING: there is no transaction in progress
2026-06-19 02:51:21,034 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:51:21,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:51:21,284 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:51:21,284 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:51:21,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:51:21,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:51:21,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jt0t.c.gcp-upstream.internal','N') ;"
2026-06-19 02:51:21,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:51:22,240 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:51:22,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:51:22,494 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:51:22,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:51:22,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:51:22,967 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:51:22,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:51:23.914 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:23.930 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:23.947 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:23.962 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:23.978 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:23.995 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:24.057 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:24.064 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:24.078 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:24.089 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:24.100 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:24.113 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:24.123 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:24.130 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:24.136 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:24.143 UTC [1661] WARNING: there is no transaction in progress
2026-06-19 02:51:24,775 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:51:24,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:51:25,023 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:51:25,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:51:25,283 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:51:25,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:51:25,543 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:51:25,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:51:25,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:51:26,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jt0t.c.gcp-upstream.internal';"
2026-06-19 02:51:26,245 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:51:26,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:51:26,502 [I] DB_PATCHES have already been applied
I20260619 02:51:26.507905 22163 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-admin:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1oh3JA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:51:26.507966 22163 mini_ranger.cc:193] Using host: 127.21.164.212
I20260619 02:51:26.511675 22163 mini_ranger.cc:227] Ranger admin URL: http://127.21.164.212:52775
Jun 19, 2026 2:51:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:51:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:51:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:51:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:51:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:51:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:51:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:51:27.510881 22163 mini_ranger.cc:160] Time spent starting Ranger: real 8.176s user 0.000s sys 0.008s
Jun 19, 2026 2:51:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:51:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:51:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:51:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:51:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:51:49.742795 22163 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.164.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.164.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/rangerkms_127.21.164.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.164.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/rangerkms_127.21.164.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.164.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.164.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.164.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/HTTP_127.21.164.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.164.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/HTTP_127.21.164.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:51:49 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837509, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:51:49.820101 22163 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:51:49.828965 22163 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:51:50.648439 22163 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:51:50.648519 22163 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-kms
I20260619 02:51:50.648535 22163 mini_ranger_kms.cc:227] Using postgres at 127.21.164.212:38943
2026-06-19 02:51:50,798 [I] DB FLAVOR :POSTGRES
2026-06-19 02:51:50,798 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:51:50,799 [I] Checking connection
2026-06-19 02:51:50,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:51:51,030 [I] connection success
2026-06-19 02:51:51,031 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:51:51,031 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:51:51,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:51:51,289 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:51:51,289 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:51:51,290 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:51:51,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.164.212:38943/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:51:51,538 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:51:52.007273 22163 mini_ranger_kms.cc:310] Created kms service
I20260619 02:51:52.222759 22163 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:51:52.266352 22163 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:51:52 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837509, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:51:52.696537 22163 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:51:52.696931 22163 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-kms:/tmp/dist-test-task1oh3JA/build/release/bin/postgresql.jar:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1oh3JA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:51:52.696983 22163 mini_ranger_kms.cc:240] Using host: 127.21.164.212
I20260619 02:51:52.700176 22163 mini_ranger_kms.cc:276] Ranger KMS PID: 1841
I20260619 02:51:52.700256 22163 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.164.212:35835
Jun 19, 2026 2:51:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:51:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:51:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:51:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:51:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:51:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 02:51:54.733943 22163 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.914s user 0.003s sys 0.130s
I20260619 02:51:54.734025 22163 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:51:54.734068 22163 mini_ranger_kms.cc:401] 127.21.164.212:35835/kms/v1/keys
Jun 19 02:51:54 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.164.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 02:51:55 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.164.212@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
Jun 19 02:51:55 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.164.212@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
Jun 19 02:51:57 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:51:57.966578 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34693
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:34693
--ranger_config_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:35835/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:58.046061 1892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:58.046285 1892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:58.046316 1892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:58.047499 1892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:51:58.047550 1892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:58.047569 1892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:58.047585 1892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:51:58.047600 1892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:51:58.049193 1892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.164.212:35835/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:34693
--ranger_config_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:51:58.049434 1892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:58.049645 1892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:51:58.054611 1892 server_base.cc:1034] running on GCE node
W20260619 02:51:58.054678 1897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:58.054652 1900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:58.054948 1898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:58.055234 1892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:58.055591 1892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:58.056722 1892 hybrid_clock.cc:648] HybridClock initialized: now 1781837518056709 us; error 29 us; skew 500 ppm
Jun 19 02:51:58 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837518, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:58.059772 1892 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:51:58.060338 1892 webserver.cc:460] Webserver started at http://127.21.164.254:40965/ using document root <none> and password file <none>
I20260619 02:51:58.060599 1892 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:58.060675 1892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:58.060843 1892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:58 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837518, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:51:58.109139 1892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "82013c2f2d7343cebc4e913b4b80c526"
format_stamp: "Formatted at 2026-06-19 02:51:58 on dist-test-slave-jt0t"
server_key: "94dc0561327ca17d351e5c0edbfbfc81"
server_key_iv: "b779aab1506392b35716ff03917a38d0"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:58.109593 1892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "82013c2f2d7343cebc4e913b4b80c526"
format_stamp: "Formatted at 2026-06-19 02:51:58 on dist-test-slave-jt0t"
server_key: "94dc0561327ca17d351e5c0edbfbfc81"
server_key_iv: "b779aab1506392b35716ff03917a38d0"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:58.111435 1892 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:51:58.127136 1907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:58.127362 1892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:51:58.127457 1892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "82013c2f2d7343cebc4e913b4b80c526"
format_stamp: "Formatted at 2026-06-19 02:51:58 on dist-test-slave-jt0t"
server_key: "94dc0561327ca17d351e5c0edbfbfc81"
server_key_iv: "b779aab1506392b35716ff03917a38d0"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:58.127552 1892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:58.155145 1892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:58.157552 1892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:58.157711 1892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:58.162788 1892 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:34693
I20260619 02:51:58.162827 1959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:34693 every 8 connection(s)
I20260619 02:51:58.163249 1892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:51:58.163998 1960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:51:58.166349 1960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526: Bootstrap starting.
I20260619 02:51:58.166931 1960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526: Neither blocks nor log segments found. Creating new log.
I20260619 02:51:58.167196 1960 log.cc:824] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526: Log is configured to *not* fsync() on all Append() calls
I20260619 02:51:58.167831 1960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526: No bootstrap required, opened a new log
I20260619 02:51:58.169997 1960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } }
I20260619 02:51:58.170152 1960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:51:58.170173 1960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82013c2f2d7343cebc4e913b4b80c526, State: Initialized, Role: FOLLOWER
I20260619 02:51:58.170308 1960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } }
I20260619 02:51:58.170377 1960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:51:58.170401 1960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:51:58.170446 1960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:51:58.170984 1960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } }
I20260619 02:51:58.171077 1960 leader_election.cc:302] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82013c2f2d7343cebc4e913b4b80c526; no voters:
I20260619 02:51:58.171237 1960 leader_election.cc:288] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:51:58.171274 1963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:51:58.171447 1963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 1 LEADER]: Becoming Leader. State: Replica: 82013c2f2d7343cebc4e913b4b80c526, State: Running, Role: LEADER
I20260619 02:51:58.171515 1960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:51:58.171617 1963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } }
I20260619 02:51:58.171990 1964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82013c2f2d7343cebc4e913b4b80c526. Latest consensus state: current_term: 1 leader_uuid: "82013c2f2d7343cebc4e913b4b80c526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } } }
I20260619 02:51:58.172060 1964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [sys.catalog]: This master's current role is: LEADER
I20260619 02:51:58.172264 1965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82013c2f2d7343cebc4e913b4b80c526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } } }
I20260619 02:51:58.172520 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 1892
I20260619 02:51:58.172614 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:51:58.172829 22163 external_mini_cluster.cc:1428] Setting key bef62f4b18568b571f347624f1d1d6ab
I20260619 02:51:58.172698 1968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:51:58.173130 1960 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:51:58.173377 1968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:51:58.173751 1965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [sys.catalog]: This master's current role is: LEADER
I20260619 02:51:58.182825 1968 catalog_manager.cc:1347] Generated new cluster ID: f70c747c5b844ef687964a3f5625ee41
I20260619 02:51:58.182911 1968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:51:58 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:51:58.189348 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:58.173440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.21.164.254:34693)
0619 02:51:58.173616 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:58.173626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:58.175880 (+ 2254us) server_negotiation.cc:407] Connection header received
0619 02:51:58.176967 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:58.176983 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:58.177117 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:58.177305 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:58.178450 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:58.179973 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:58.181104 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:58.181757 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:58.181879 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:58.184717 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:58.184722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:58.184722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:58.184736 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:58.187245 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:58.187680 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:58.187684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:58.187685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:58.187799 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:58.187992 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:58.187992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:58.187993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:58.188336 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:58.188475 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:58.188953 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 02:51:58.189088 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260619 02:51:58.196131 1968 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:51:58.196970 1968 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:51:58.201498 1968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526: Generated new TSK 0
I20260619 02:51:58.201756 1968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:51:58.987039 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:35835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34693
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:59.072180 2005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:59.072386 2005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:59.072408 2005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:59.073710 2005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:59.073758 2005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:59.073784 2005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:51:59.075443 2005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.164.212:35835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:51:59.075793 2005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:59.076045 2005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:59.081193 2010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:59.081104 2011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:59.081403 2005 server_base.cc:1034] running on GCE node
W20260619 02:51:59.081104 2013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:59.081880 2005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:59.082159 2005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:59.083304 2005 hybrid_clock.cc:648] HybridClock initialized: now 1781837519083290 us; error 34 us; skew 500 ppm
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:59.085907 2005 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:51:59.086393 2005 webserver.cc:460] Webserver started at http://127.21.164.193:36949/ using document root <none> and password file <none>
I20260619 02:51:59.086618 2005 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:59.086668 2005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:59.086794 2005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:51:59.101732 2005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "bbef088393c541b5a1ee29e8a6cf0f23"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "aadb104ad2bfaa3629427a17965fe1dc"
server_key_iv: "4ca3c40128a21168580d38b7042aa378"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:59.102269 2005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "bbef088393c541b5a1ee29e8a6cf0f23"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "aadb104ad2bfaa3629427a17965fe1dc"
server_key_iv: "4ca3c40128a21168580d38b7042aa378"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:59.104432 2005 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:51:59.119127 2020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:59.119374 2005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:51:59.119486 2005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "bbef088393c541b5a1ee29e8a6cf0f23"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "aadb104ad2bfaa3629427a17965fe1dc"
server_key_iv: "4ca3c40128a21168580d38b7042aa378"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:59.119599 2005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:59.131564 2005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:59.134020 2005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:59.134245 2005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:59.134548 2005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:51:59.134997 2005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:51:59.135093 2005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:59.135157 2005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:51:59.135190 2005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:59.141772 2005 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:38823
I20260619 02:51:59.142299 2005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:51:59.142541 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 2005
I20260619 02:51:59.142637 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:51:59.142802 22163 external_mini_cluster.cc:1428] Setting key 80f13a60f895801c0368503dbc75cbf6
I20260619 02:51:59.144233 2134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:38823 every 8 connection(s)
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:51:59.152364 2137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:59.143124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:39637 (local address 127.21.164.254:34693)
0619 02:51:59.144553 (+ 1429us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:59.144555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:59.144562 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:51:59.144586 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:59.144586 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:59.144599 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:59.144689 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:59.145500 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.146315 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.147333 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.147538 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.147598 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:59.149658 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:59.149660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:59.149660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:59.149668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:59.151021 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:59.151407 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:59.151409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:59.151409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:59.151473 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:59.151706 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:59.151707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:59.151707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:59.151822 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:59.151878 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:59.152170 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 02:51:59.152262 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1395,"thread_start_us":84,"threads_started":1}
I20260619 02:51:59.152526 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:59.143130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34693 (local address 127.21.164.193:39637)
0619 02:51:59.143484 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:59.143732 (+ 248us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:59.144359 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:59.144830 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:59.144835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:59.145079 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:51:59.145376 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:59.145383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.146416 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.146417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:59.147241 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:59.147244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.147625 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.147626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:59.147758 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:59.148268 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:51:59.148277 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:51:59.149566 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:51:59.151097 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:59.151104 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:59.151107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:59.151339 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:59.151529 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:59.151530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:59.151530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:59.151651 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:59.151876 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:51:59.151883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:51:59.152050 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 02:51:59.152143 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:51:59.152293 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
I20260619 02:51:59.153122 2135 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34693
I20260619 02:51:59.153227 2135 heartbeater.cc:464] Registering TS with master...
I20260619 02:51:59.153412 2135 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
I20260619 02:51:59.153932 1924 ts_manager.cc:194] Registered new tserver with Master: bbef088393c541b5a1ee29e8a6cf0f23 (127.21.164.193:38823)
I20260619 02:51:59.154727 1924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:39637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:51:59.180258 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:35835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34693
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:59.301319 2142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:59.301671 2142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:59.301761 2142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:59.303561 2142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:59.303685 2142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:59.303767 2142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:51:59.306171 2142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.164.212:35835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:51:59.306702 2142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:59.307053 2142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:59.321909 2150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:59.322027 2149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:59.322638 2152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:59.322892 2142 server_base.cc:1034] running on GCE node
I20260619 02:51:59.323276 2142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:59.323676 2142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:59.324916 2142 hybrid_clock.cc:648] HybridClock initialized: now 1781837519324884 us; error 45 us; skew 500 ppm
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:59.328426 2142 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:51:59.329001 2142 webserver.cc:460] Webserver started at http://127.21.164.194:41003/ using document root <none> and password file <none>
I20260619 02:51:59.329259 2142 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:59.329311 2142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:59.329445 2142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:51:59.346033 2142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "565bc00f84ec4fb41bce4253d15e4d36"
server_key_iv: "b872159b9b9e8607bc509d412a03de8a"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:59.346505 2142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "565bc00f84ec4fb41bce4253d15e4d36"
server_key_iv: "b872159b9b9e8607bc509d412a03de8a"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:59.353821 2142 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:59.371253 2159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:59.372555 2142 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260619 02:51:59.372725 2142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "565bc00f84ec4fb41bce4253d15e4d36"
server_key_iv: "b872159b9b9e8607bc509d412a03de8a"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:59.372830 2142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:59.387813 2142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:59.390391 2142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:59.390622 2142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:59.390936 2142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:51:59.391357 2142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:51:59.391445 2142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:59.391512 2142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:51:59.391567 2142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:59.407372 2142 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:37829
I20260619 02:51:59.407959 2142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:51:59.411284 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 2142
I20260619 02:51:59.411427 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:51:59.411655 22163 external_mini_cluster.cc:1428] Setting key 7c71ea25aec6659e31e46879fb74671c
I20260619 02:51:59.416396 2272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:37829 every 8 connection(s)
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:51:59.433801 2137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:59.417814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:44343 (local address 127.21.164.254:34693)
0619 02:51:59.417980 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:59.417983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:59.419508 (+ 1525us) server_negotiation.cc:407] Connection header received
0619 02:51:59.420225 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:59.420227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:59.420243 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:59.420273 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:59.421174 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.421865 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.423469 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.423788 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.423830 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:59.430379 (+ 6549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:59.430381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:59.430382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:59.430391 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:59.432021 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:59.432581 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:59.432582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:59.432583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:59.432644 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:59.432997 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:59.432998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:59.432999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:59.433143 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:59.433188 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:59.433599 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 02:51:59.433701 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:51:59.434636 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:59.417509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34693 (local address 127.21.164.194:44343)
0619 02:51:59.419343 (+ 1834us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:59.419405 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:59.420094 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:59.420464 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:59.420469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:59.420756 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:51:59.421060 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:59.421069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.421991 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.421992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:59.423350 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:59.423353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.423970 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.423971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:59.424112 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:59.424751 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:51:59.424769 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:51:59.430242 (+ 5473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:51:59.432148 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:59.432159 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:59.432163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:59.432488 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:59.432748 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:59.432749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:59.432749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:59.432920 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:59.433232 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:51:59.433239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:51:59.433432 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 02:51:59.434191 (+ 759us) client_negotiation.cc:240] Negotiation successful
0619 02:51:59.434383 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1780,"thread_start_us":107,"threads_started":1}
I20260619 02:51:59.435354 2273 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34693
I20260619 02:51:59.435459 2273 heartbeater.cc:464] Registering TS with master...
I20260619 02:51:59.435688 2273 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
I20260619 02:51:59.436167 1924 ts_manager.cc:194] Registered new tserver with Master: f4f0c4ee0b1a46f8bb8f22598de54eda (127.21.164.194:37829)
I20260619 02:51:59.436702 1924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:44343
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:51:59.459331 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:35835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34693
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:59.585592 2279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:59.585943 2279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:59.586027 2279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:59.587806 2279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:59.588063 2279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:59.588176 2279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:51:59.590549 2279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.164.212:35835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:51:59.591054 2279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:59.591432 2279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:59.598392 2287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:59.598857 2284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:59.599618 2285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:59.599944 2279 server_base.cc:1034] running on GCE node
I20260619 02:51:59.600184 2279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:59.600485 2279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:59.601661 2279 hybrid_clock.cc:648] HybridClock initialized: now 1781837519601595 us; error 77 us; skew 500 ppm
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:59.605250 2279 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:51:59.605871 2279 webserver.cc:460] Webserver started at http://127.21.164.195:40117/ using document root <none> and password file <none>
I20260619 02:51:59.606140 2279 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:59.606199 2279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:51:59.606333 2279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:51:59.629473 2279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "2f0662f15e964f078578118087757d37"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "38875cc3a0ce14ed238e4b67109f24de"
server_key_iv: "ee0852685f8e4e9de60c6c798eae550e"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:59.631925 2279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "2f0662f15e964f078578118087757d37"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "38875cc3a0ce14ed238e4b67109f24de"
server_key_iv: "ee0852685f8e4e9de60c6c798eae550e"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:59.635426 2279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:51:59.658952 2294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:59.659216 2279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:51:59.659332 2279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "2f0662f15e964f078578118087757d37"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "38875cc3a0ce14ed238e4b67109f24de"
server_key_iv: "ee0852685f8e4e9de60c6c798eae550e"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:59.659436 2279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:59.679323 2279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:59.690335 2279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:59.690570 2279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:59.690873 2279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:51:59.691330 2279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:51:59.691421 2279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:59.691490 2279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:51:59.691545 2279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:59.697988 2279 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:36697
I20260619 02:51:59.698530 2279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:51:59.701234 2407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:36697 every 8 connection(s)
I20260619 02:51:59.705571 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 2279
I20260619 02:51:59.705680 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:51:59.705855 22163 external_mini_cluster.cc:1428] Setting key 12ad76e98ae43ec709a4614d3ab50ef4
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:51:59.714745 2137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:59.702236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:57251 (local address 127.21.164.254:34693)
0619 02:51:59.702284 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:51:59.702286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:51:59.702622 (+ 336us) server_negotiation.cc:407] Connection header received
0619 02:51:59.703420 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:51:59.703422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:51:59.703435 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:51:59.703464 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:51:59.704313 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.705145 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.708196 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.708507 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.708552 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:59.711385 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:51:59.711387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:51:59.711387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:51:59.711396 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:51:59.713153 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:59.713650 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:59.713652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:59.713652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:59.713713 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:51:59.713998 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:51:59.713999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:51:59.713999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:51:59.714142 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:51:59.714182 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 02:51:59.714557 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 02:51:59.714642 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:51:59.715548 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:51:59.701950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34693 (local address 127.21.164.195:57251)
0619 02:51:59.702489 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 02:51:59.702537 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:51:59.703293 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:51:59.703653 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:51:59.703657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:51:59.703915 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:51:59.704202 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:59.704208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.705257 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.705258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:59.708083 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:51:59.708087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:51:59.708701 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:51:59.708702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:51:59.708863 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:51:59.709529 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:51:59.709540 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:51:59.711237 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:51:59.713254 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:59.713262 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:59.713265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:59.713563 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:59.713779 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:51:59.713779 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:51:59.713780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:51:59.713929 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:51:59.714227 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:51:59.714232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:51:59.714402 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 02:51:59.715128 (+ 726us) client_negotiation.cc:240] Negotiation successful
0619 02:51:59.715301 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":79,"threads_started":1}
I20260619 02:51:59.716200 2408 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34693
I20260619 02:51:59.716295 2408 heartbeater.cc:464] Registering TS with master...
I20260619 02:51:59.716480 2408 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
I20260619 02:51:59.717154 1924 ts_manager.cc:194] Registered new tserver with Master: 2f0662f15e964f078578118087757d37 (127.21.164.195:36697)
I20260619 02:51:59.717626 1924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:57251
I20260619 02:51:59.718952 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:51:59.719576 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 2005
I20260619 02:51:59.724829 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 2142
I20260619 02:51:59.730487 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 2279
I20260619 02:51:59.736404 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 1892
I20260619 02:51:59.741962 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34693
--webserver_interface=127.21.164.254
--webserver_port=40965
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:34693
--ranger_config_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:35835/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.164.212:35835/kms with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:51:59.823909 2411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:51:59.824123 2411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:51:59.824144 2411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:51:59.825321 2411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:51:59.825363 2411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:51:59.825376 2411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:51:59.825387 2411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:51:59.825398 2411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:51:59.826984 2411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.164.212:35835/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:34693
--ranger_config_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=40965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:51:59.827201 2411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:51:59.827405 2411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:51:59.831976 2416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:59.832000 2419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:51:59.831969 2417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:51:59.832090 2411 server_base.cc:1034] running on GCE node
I20260619 02:51:59.832365 2411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:51:59.832670 2411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:51:59.833844 2411 hybrid_clock.cc:648] HybridClock initialized: now 1781837519833825 us; error 30 us; skew 500 ppm
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:51:59.836338 2411 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:51:59.836850 2411 webserver.cc:460] Webserver started at http://127.21.164.254:40965/ using document root <none> and password file <none>
I20260619 02:51:59.837050 2411 fs_manager.cc:359] Metadata directory not provided
I20260619 02:51:59.837102 2411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:51:59 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:51:59.914109 2411 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:51:59.915012 2427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:51:59.915218 2411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:51:59.915336 2411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "82013c2f2d7343cebc4e913b4b80c526"
format_stamp: "Formatted at 2026-06-19 02:51:58 on dist-test-slave-jt0t"
server_key: "94dc0561327ca17d351e5c0edbfbfc81"
server_key_iv: "b779aab1506392b35716ff03917a38d0"
server_key_version: "kuduclusterkey@0"
I20260619 02:51:59.915819 2411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:51:59.929770 2411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:51:59.931627 2411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:51:59.931751 2411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:51:59.935678 2411 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:34693
I20260619 02:51:59.935717 2479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:34693 every 8 connection(s)
I20260619 02:51:59.936076 2411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:51:59.937376 2480 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:51:59.937868 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 2411
I20260619 02:51:59.938081 2480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526: Bootstrap starting.
I20260619 02:51:59.938135 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:35835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:38823
--local_ip_for_outbound_sockets=127.21.164.193
--tserver_master_addrs=127.21.164.254:34693
--webserver_port=36949
--webserver_interface=127.21.164.193
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.164.212:35835/kms with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:51:59.945770 2480 log.cc:824] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526: Log is configured to *not* fsync() on all Append() calls
I20260619 02:51:59.947454 2480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:51:59.947706 2480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526: Bootstrap complete.
I20260619 02:51:59.949944 2480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } }
I20260619 02:51:59.950187 2480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82013c2f2d7343cebc4e913b4b80c526, State: Initialized, Role: FOLLOWER
I20260619 02:51:59.950362 2480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } }
I20260619 02:51:59.950441 2480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:51:59.950489 2480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:51:59.950534 2480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:51:59.951540 2480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } }
I20260619 02:51:59.951644 2480 leader_election.cc:302] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82013c2f2d7343cebc4e913b4b80c526; no voters:
I20260619 02:51:59.951813 2480 leader_election.cc:288] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:51:59.951884 2484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:51:59.952064 2480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:51:59.952090 2484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [term 2 LEADER]: Becoming Leader. State: Replica: 82013c2f2d7343cebc4e913b4b80c526, State: Running, Role: LEADER
I20260619 02:51:59.952226 2484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } }
I20260619 02:51:59.952505 2485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "82013c2f2d7343cebc4e913b4b80c526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } } }
I20260619 02:51:59.952672 2485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [sys.catalog]: This master's current role is: LEADER
I20260619 02:51:59.952868 2486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82013c2f2d7343cebc4e913b4b80c526. Latest consensus state: current_term: 2 leader_uuid: "82013c2f2d7343cebc4e913b4b80c526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82013c2f2d7343cebc4e913b4b80c526" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34693 } } }
I20260619 02:51:59.952956 2486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526 [sys.catalog]: This master's current role is: LEADER
I20260619 02:51:59.953091 2489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:51:59.953583 2489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:51:59.953706 2489 catalog_manager.cc:1259] Loaded cluster ID: f70c747c5b844ef687964a3f5625ee41
I20260619 02:51:59.953739 2489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:51:59.953960 2480 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:51:59.957072 2489 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:51:59.957628 2489 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526: Loaded TSK: 0
I20260619 02:51:59.957918 2489 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:52:00.025558 2482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:00.025758 2482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:00.025790 2482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:00.026959 2482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:00.027004 2482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:00.027037 2482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:52:00.028949 2482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.164.212:35835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:38823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=36949
--tserver_master_addrs=127.21.164.254:34693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:00.029429 2482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:00.029762 2482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:00.035223 2519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:00.035239 2516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:00.035252 2517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:00.035604 2482 server_base.cc:1034] running on GCE node
I20260619 02:52:00.035964 2482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:00.036265 2482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:00.037442 2482 hybrid_clock.cc:648] HybridClock initialized: now 1781837520037412 us; error 42 us; skew 500 ppm
Jun 19 02:52:00 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:00.040813 2482 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:52:00.041399 2482 webserver.cc:460] Webserver started at http://127.21.164.193:36949/ using document root <none> and password file <none>
I20260619 02:52:00.041677 2482 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:00.041803 2482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:52:00 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:52:00.069803 2482 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:52:00.070907 2528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:00.071142 2482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:00.071273 2482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "bbef088393c541b5a1ee29e8a6cf0f23"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "aadb104ad2bfaa3629427a17965fe1dc"
server_key_iv: "4ca3c40128a21168580d38b7042aa378"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:00.071760 2482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:00.108318 2482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:00.110747 2482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:00.110971 2482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:00.111292 2482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:00.111714 2482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:00.111791 2482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:00.111850 2482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:00.111891 2482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:00.118690 2482 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:38823
I20260619 02:52:00.119287 2482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:00.119699 2641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:38823 every 8 connection(s)
I20260619 02:52:00.123776 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 2482
I20260619 02:52:00.124059 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:35835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:37829
--local_ip_for_outbound_sockets=127.21.164.194
--tserver_master_addrs=127.21.164.254:34693
--webserver_port=41003
--webserver_interface=127.21.164.194
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.164.212:35835/kms with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:52:00 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:00.133176 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:00.121029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:49613 (local address 127.21.164.254:34693)
0619 02:52:00.121254 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:00.121265 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:00.121426 (+ 161us) server_negotiation.cc:407] Connection header received
0619 02:52:00.122276 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:00.122284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:00.122347 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:00.122487 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:00.123262 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.124714 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.126311 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.126903 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.129639 (+ 2736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:00.129734 (+ 95us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:00.129737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:00.129738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:00.129747 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:00.131422 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:00.131841 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:00.131844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:00.131845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:00.131936 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:00.132177 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:00.132177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:00.132178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:00.132421 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:00.132495 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:00.132831 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 02:52:00.132945 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
I20260619 02:52:00.133189 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:00.120604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34693 (local address 127.21.164.193:49613)
0619 02:52:00.121266 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:00.121317 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:00.121959 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:00.122560 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:00.122568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:00.122813 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:00.123088 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:00.123093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.124865 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.124867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:00.126190 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:00.126194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.127057 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.127059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:00.127207 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:00.127813 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:00.127825 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:00.129483 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:00.131520 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:00.131529 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:00.131531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:00.131760 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:00.132002 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:00.132003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:00.132004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:00.132117 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:00.132497 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:00.132503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:00.132633 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 02:52:00.132767 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 02:52:00.132945 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":117,"threads_started":1}
I20260619 02:52:00.134021 2642 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34693
I20260619 02:52:00.134119 2642 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:00.134297 2642 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
W20260619 02:52:00.135108 2642 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:34693 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:52:00.135187 2642 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:00.135254 2642 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
I20260619 02:52:00.135550 2642 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:00.135627 2642 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
W20260619 02:52:00.135999 2642 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:52:00.247220 2646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:00.247527 2646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:00.247574 2646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:00.249487 2646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:00.249599 2646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:00.249678 2646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:52:00.252140 2646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.164.212:35835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:37829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=41003
--tserver_master_addrs=127.21.164.254:34693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:00.252558 2646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:00.252940 2646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:00.261904 2654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:00.261911 2652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:00.261909 2651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:00.262130 2646 server_base.cc:1034] running on GCE node
I20260619 02:52:00.262953 2646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:00.263216 2646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:00.264393 2646 hybrid_clock.cc:648] HybridClock initialized: now 1781837520264365 us; error 42 us; skew 500 ppm
Jun 19 02:52:00 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:00.267577 2646 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:52:00.268154 2646 webserver.cc:460] Webserver started at http://127.21.164.194:41003/ using document root <none> and password file <none>
I20260619 02:52:00.268378 2646 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:00.268433 2646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:52:00 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:52:00.293107 2646 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:00.294236 2662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:00.294492 2646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:00.294600 2646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "565bc00f84ec4fb41bce4253d15e4d36"
server_key_iv: "b872159b9b9e8607bc509d412a03de8a"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:00.295091 2646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:00.317802 2646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:00.320819 2646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:00.321005 2646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:00.321308 2646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:00.321727 2646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:00.321782 2646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:00.321815 2646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:00.321884 2646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:00.329517 2646 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:37829
I20260619 02:52:00.330137 2646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:52:00.333187 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 2646
I20260619 02:52:00.333487 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.164.212:35835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:36697
--local_ip_for_outbound_sockets=127.21.164.195
--tserver_master_addrs=127.21.164.254:34693
--webserver_port=40117
--webserver_interface=127.21.164.195
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.164.212:35835/kms with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:00.331530 2775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:37829 every 8 connection(s)
Jun 19 02:52:00 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:00.348873 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:00.331366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:34245 (local address 127.21.164.254:34693)
0619 02:52:00.331423 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:00.331425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:00.332068 (+ 643us) server_negotiation.cc:407] Connection header received
0619 02:52:00.332818 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:00.332819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:00.332836 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:00.332865 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:00.335845 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.336624 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.338242 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.338573 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.338655 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:00.345215 (+ 6560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:00.345218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:00.345218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:00.345229 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:00.347035 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:00.347626 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:00.347628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:00.347628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:00.347694 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:00.348016 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:00.348017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:00.348017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:00.348168 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:00.348215 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:00.348630 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 02:52:00.348769 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:52:00.349709 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:00.331056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34693 (local address 127.21.164.194:34245)
0619 02:52:00.331899 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:00.331951 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:00.332683 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:00.335120 (+ 2437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:00.335126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:00.335373 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:00.335701 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:00.335711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.336772 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.336773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:00.338132 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:00.338136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.339846 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.339848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:00.340017 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:00.340632 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:00.340667 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:00.345051 (+ 4384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:00.347143 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:00.347155 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:00.347158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:00.347528 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:00.347772 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:00.347772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:00.347773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:00.347929 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:00.348263 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:00.348270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:00.348467 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 02:52:00.349252 (+ 785us) client_negotiation.cc:240] Negotiation successful
0619 02:52:00.349455 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":78,"threads_started":1}
I20260619 02:52:00.350476 2776 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34693
I20260619 02:52:00.350590 2776 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:00.350821 2776 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
W20260619 02:52:00.351573 2776 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:34693 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:52:00.351648 2776 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:00.351711 2776 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
I20260619 02:52:00.352070 2776 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:00.352142 2776 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
W20260619 02:52:00.352483 2776 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:52:00.462919 2779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:00.463253 2779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:00.463299 2779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:00.465231 2779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:00.465346 2779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:00.465422 2779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:52:00.467772 2779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.164.212:35835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:36697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=40117
--tserver_master_addrs=127.21.164.254:34693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:00.468256 2779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:00.468672 2779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:00.475068 2788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:00.475075 2785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:00.475073 2786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:00.475765 2779 server_base.cc:1034] running on GCE node
I20260619 02:52:00.476023 2779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:00.476336 2779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:00.477526 2779 hybrid_clock.cc:648] HybridClock initialized: now 1781837520477501 us; error 38 us; skew 500 ppm
Jun 19 02:52:00 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:00.480518 2779 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:52:00.481022 2779 webserver.cc:460] Webserver started at http://127.21.164.195:40117/ using document root <none> and password file <none>
I20260619 02:52:00.481241 2779 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:00.481345 2779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:52:00 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:52:00.497237 2779 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:52:00.501072 2796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:00.501240 2779 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 02:52:00.501325 2779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "2f0662f15e964f078578118087757d37"
format_stamp: "Formatted at 2026-06-19 02:51:59 on dist-test-slave-jt0t"
server_key: "38875cc3a0ce14ed238e4b67109f24de"
server_key_iv: "ee0852685f8e4e9de60c6c798eae550e"
server_key_version: "kuduclusterkey@0"
I20260619 02:52:00.501830 2779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:00.518666 2779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:00.520639 2779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:00.520828 2779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:00.521040 2779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:00.521395 2779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:00.521435 2779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:00.521466 2779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:00.521503 2779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:00.527872 2779 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:36697
I20260619 02:52:00.527951 2909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:36697 every 8 connection(s)
I20260619 02:52:00.528311 2779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:52:00 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:00.536250 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 2779
I20260619 02:52:00.539180 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:00.529067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:40331 (local address 127.21.164.254:34693)
0619 02:52:00.529136 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:00.529138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:00.529302 (+ 164us) server_negotiation.cc:407] Connection header received
0619 02:52:00.529790 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:00.529792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:00.529813 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:00.529845 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:00.530611 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.531347 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.532329 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.532530 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.532585 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:00.535101 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:00.535103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:00.535103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:00.535113 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:00.537360 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:00.537981 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:00.537983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:00.537983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:00.538067 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:00.538372 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:00.538373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:00.538373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:00.538530 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:00.538579 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:00.538986 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 02:52:00.539081 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:52:00.540445 2912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:00.529019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34693 (local address 127.21.164.195:40331)
0619 02:52:00.529168 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:00.529210 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:00.529679 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:00.530041 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:00.530046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:00.530263 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:00.530520 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:00.530527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.531436 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.531437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:00.532243 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:00.532246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.532703 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.532705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:00.532848 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:00.533460 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:00.533470 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:00.534997 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:00.537476 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:00.537488 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:00.537491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:00.537881 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:00.538138 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:00.538138 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:00.538139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:00.538293 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:00.538629 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:00.538636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:00.538832 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 02:52:00.538957 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 02:52:00.539137 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260619 02:52:00.541191 2910 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34693
I20260619 02:52:00.541355 2910 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:00.541635 2910 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
W20260619 02:52:00.542459 2910 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:34693 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:52:00.542533 2910 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:00.542594 2910 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
I20260619 02:52:00.542953 2910 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:00.543037 2910 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
W20260619 02:52:00.543401 2910 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:52:00.546658 2913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:00.536953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.21.164.254:34693)
0619 02:52:00.537098 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:00.537101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:00.537980 (+ 879us) server_negotiation.cc:407] Connection header received
0619 02:52:00.538007 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:00.538008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:00.538025 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:00.538087 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:00.539490 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.540228 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.541856 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:00.542181 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:00.542258 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:00.543714 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:00.543716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:00.543716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:00.543724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:00.545546 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:00.545904 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:00.545906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:00.545906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:00.545971 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:00.546142 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:00.546143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:00.546144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:00.546261 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:00.546433 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:00.546468 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 02:52:00.546552 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
Jun 19 02:52:01 dist-test-slave-jt0t krb5kdc[1448](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837521, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:01.136188 2642 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:01.136351 2642 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
I20260619 02:52:01.136806 2444 ts_manager.cc:194] Registered new tserver with Master: bbef088393c541b5a1ee29e8a6cf0f23 (127.21.164.193:38823)
I20260619 02:52:01.137964 2444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:49613
I20260619 02:52:01.352681 2776 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:01.352852 2776 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
I20260619 02:52:01.353313 2444 ts_manager.cc:194] Registered new tserver with Master: f4f0c4ee0b1a46f8bb8f22598de54eda (127.21.164.194:37829)
I20260619 02:52:01.353824 2444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:34245
Jun 19 02:52:01 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837521, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:52:01.543632 2910 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:01.543790 2910 heartbeater.cc:511] Master 127.21.164.254:34693 requested a full tablet report, sending...
I20260619 02:52:01.544126 2444 ts_manager.cc:194] Registered new tserver with Master: 2f0662f15e964f078578118087757d37 (127.21.164.195:36697)
I20260619 02:52:01.544467 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:52:01.544620 2444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:40331
I20260619 02:52:01.553341 2931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.545447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.21.164.254:34693)
0619 02:52:01.545671 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:01.545674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:01.545807 (+ 133us) server_negotiation.cc:407] Connection header received
0619 02:52:01.545873 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:01.545873 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:01.545885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:01.545956 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:01.546654 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.547385 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.548426 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.548758 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.548822 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.550255 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:01.550257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:01.550257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:01.550265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:01.552025 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:01.552434 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:01.552435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:01.552435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:01.552495 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:01.552762 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:01.552763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:01.552763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:01.552902 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:01.552997 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:01.553152 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 02:52:01.553233 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":67,"threads_started":1}
I20260619 02:52:01.555997 2444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:52:01.558243 2444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:52:01.569139 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.561373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:36697 (local address 127.0.0.1:35290)
0619 02:52:01.562281 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:01.562323 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:01.562383 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:01.565201 (+ 2818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:01.565202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:01.565205 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:01.565264 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:01.565266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.566529 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.566530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:01.567825 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:01.567829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.568836 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.568837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:01.568920 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.568935 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 02:52:01.568995 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:52:01.569046 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":29,"thread_start_us":82,"threads_started":1}
I20260619 02:52:01.569487 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.561388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:37829 (local address 127.0.0.1:48416)
0619 02:52:01.564769 (+ 3381us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:01.564781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:01.564845 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:01.565072 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:01.565075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:01.565078 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:01.565171 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:01.565173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.566855 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.566856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:01.567987 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:01.567990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.568982 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.568983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:01.569045 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.569050 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:52:01.569077 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 02:52:01.569410 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3359,"thread_start_us":82,"threads_started":1}
I20260619 02:52:01.569676 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.561635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.21.164.195:36697)
0619 02:52:01.564978 (+ 3343us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:01.564983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:01.564994 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:52:01.565028 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:01.565030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:01.565045 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:01.565117 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:01.565357 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.566432 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.567914 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.568629 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.568750 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.568812 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:01.569561 (+ 749us) server_negotiation.cc:299] Negotiation successful
0619 02:52:01.569604 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3303,"thread_start_us":60,"threads_started":1}
I20260619 02:52:01.570036 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.561642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.21.164.194:37829)
0619 02:52:01.562214 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:01.562220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:01.564861 (+ 2641us) server_negotiation.cc:407] Connection header received
0619 02:52:01.564918 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:01.564922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:01.564940 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:01.565255 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:01.565343 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.566762 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.568091 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.568884 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.569812 (+ 928us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.569878 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:01.569914 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 02:52:01.569957 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":49,"threads_started":1}
I20260619 02:52:01.570465 2844 tablet_service.cc:1467] Processing CreateTablet for tablet 703920f8492746b0bb53a583c6a80fdb (DEFAULT_TABLE table=test-table [id=f8163d5a746345f880c14ea5db66a079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:01.570842 2844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703920f8492746b0bb53a583c6a80fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:01.572474 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.562026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.21.164.193:38823)
0619 02:52:01.562273 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:01.562277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:01.568814 (+ 6537us) server_negotiation.cc:407] Connection header received
0619 02:52:01.568942 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:01.568946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:01.568965 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:01.569022 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:01.569249 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.570247 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.571237 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.572002 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.572077 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.572137 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:01.572347 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 02:52:01.572392 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":50,"threads_started":1}
I20260619 02:52:01.573211 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.561832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:38823 (local address 127.0.0.1:60102)
0619 02:52:01.568725 (+ 6893us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:01.568737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:01.568848 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:01.569137 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:01.569138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:01.569139 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:01.569175 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:01.569176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.570343 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.570344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:01.571150 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:01.571153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.572176 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.572177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:01.572242 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.572248 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:52:01.573079 (+ 831us) client_negotiation.cc:240] Negotiation successful
0619 02:52:01.573133 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6873,"thread_start_us":70,"threads_started":1}
I20260619 02:52:01.574396 2943 tablet_bootstrap.cc:492] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37: Bootstrap starting.
I20260619 02:52:01.574903 2695 tablet_service.cc:1467] Processing CreateTablet for tablet 703920f8492746b0bb53a583c6a80fdb (DEFAULT_TABLE table=test-table [id=f8163d5a746345f880c14ea5db66a079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:01.575155 2943 tablet_bootstrap.cc:654] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:01.575456 2943 log.cc:824] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:01.575881 2695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703920f8492746b0bb53a583c6a80fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:01.576231 2576 tablet_service.cc:1467] Processing CreateTablet for tablet 703920f8492746b0bb53a583c6a80fdb (DEFAULT_TABLE table=test-table [id=f8163d5a746345f880c14ea5db66a079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:01.576576 2576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 703920f8492746b0bb53a583c6a80fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:01.579715 2945 tablet_bootstrap.cc:492] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda: Bootstrap starting.
I20260619 02:52:01.580533 2945 tablet_bootstrap.cc:654] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:01.580817 2946 tablet_bootstrap.cc:492] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23: Bootstrap starting.
I20260619 02:52:01.580947 2945 log.cc:824] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:01.581439 2946 tablet_bootstrap.cc:654] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:01.581700 2945 tablet_bootstrap.cc:492] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda: No bootstrap required, opened a new log
I20260619 02:52:01.581755 2945 ts_tablet_manager.cc:1397] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:52:01.581782 2946 log.cc:824] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:01.582746 2943 tablet_bootstrap.cc:492] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37: No bootstrap required, opened a new log
I20260619 02:52:01.582814 2943 ts_tablet_manager.cc:1397] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260619 02:52:01.584309 2945 raft_consensus.cc:348] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0662f15e964f078578118087757d37" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 36697 } } peers { permanent_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37829 } } peers { permanent_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38823 } }
I20260619 02:52:01.584499 2945 raft_consensus.cc:374] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:01.584532 2945 raft_consensus.cc:729] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f0c4ee0b1a46f8bb8f22598de54eda, State: Initialized, Role: FOLLOWER
I20260619 02:52:01.584887 2945 consensus_queue.cc:260] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0662f15e964f078578118087757d37" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 36697 } } peers { permanent_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37829 } } peers { permanent_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38823 } }
I20260619 02:52:01.585249 2945 ts_tablet_manager.cc:1428] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 02:52:01.585414 2776 heartbeater.cc:503] Master 127.21.164.254:34693 was elected leader, sending a full tablet report...
I20260619 02:52:01.586057 2943 raft_consensus.cc:348] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0662f15e964f078578118087757d37" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 36697 } } peers { permanent_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37829 } } peers { permanent_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38823 } }
I20260619 02:52:01.586217 2943 raft_consensus.cc:374] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:01.586261 2943 raft_consensus.cc:729] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0662f15e964f078578118087757d37, State: Initialized, Role: FOLLOWER
I20260619 02:52:01.586373 2943 consensus_queue.cc:260] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0662f15e964f078578118087757d37" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 36697 } } peers { permanent_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37829 } } peers { permanent_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38823 } }
I20260619 02:52:01.586622 2943 ts_tablet_manager.cc:1428] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 02:52:01.586745 2910 heartbeater.cc:503] Master 127.21.164.254:34693 was elected leader, sending a full tablet report...
I20260619 02:52:01.589288 2946 tablet_bootstrap.cc:492] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23: No bootstrap required, opened a new log
I20260619 02:52:01.589378 2946 ts_tablet_manager.cc:1397] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260619 02:52:01.591377 2946 raft_consensus.cc:348] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0662f15e964f078578118087757d37" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 36697 } } peers { permanent_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37829 } } peers { permanent_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38823 } }
I20260619 02:52:01.591547 2946 raft_consensus.cc:374] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:01.591575 2946 raft_consensus.cc:729] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbef088393c541b5a1ee29e8a6cf0f23, State: Initialized, Role: FOLLOWER
I20260619 02:52:01.591717 2946 consensus_queue.cc:260] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0662f15e964f078578118087757d37" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 36697 } } peers { permanent_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37829 } } peers { permanent_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38823 } }
I20260619 02:52:01.591962 2946 ts_tablet_manager.cc:1428] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:52:01.592067 2642 heartbeater.cc:503] Master 127.21.164.254:34693 was elected leader, sending a full tablet report...
W20260619 02:52:01.620939 2643 tablet.cc:2369] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 02:52:01 dist-test-slave-jt0t krb5kdc[1448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837521, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for HTTP/127.21.164.212@KRBTEST.COM
I20260619 02:52:01.696449 2951 raft_consensus.cc:482] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:01.696594 2951 raft_consensus.cc:504] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0662f15e964f078578118087757d37" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 36697 } } peers { permanent_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37829 } } peers { permanent_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38823 } }
I20260619 02:52:01.697044 2951 leader_election.cc:288] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f0662f15e964f078578118087757d37 (127.21.164.195:36697), f4f0c4ee0b1a46f8bb8f22598de54eda (127.21.164.194:37829)
I20260619 02:52:01.701608 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.697380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:50701 (local address 127.21.164.195:36697)
0619 02:52:01.697445 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:01.697447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:01.697977 (+ 530us) server_negotiation.cc:407] Connection header received
0619 02:52:01.698046 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:01.698047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:01.698058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:01.698122 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:01.698351 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.699122 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.700596 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.701181 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.701232 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.701241 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:01.701485 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 02:52:01.701535 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:52:01.701876 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.697170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:36697 (local address 127.21.164.193:50701)
0619 02:52:01.697919 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:01.697930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:01.697989 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:01.698182 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:01.698183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:01.698186 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:01.698270 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:01.698273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.699221 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.699222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:01.700493 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:01.700496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.701325 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.701327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:01.701396 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.701400 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:52:01.701738 (+ 338us) client_negotiation.cc:240] Negotiation successful
0619 02:52:01.701782 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":95,"threads_started":1}
I20260619 02:52:01.702178 2864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703920f8492746b0bb53a583c6a80fdb" candidate_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0662f15e964f078578118087757d37" is_pre_election: true
I20260619 02:52:01.702286 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.697436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:56409 (local address 127.21.164.194:37829)
0619 02:52:01.697478 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:01.697480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:01.698119 (+ 639us) server_negotiation.cc:407] Connection header received
0619 02:52:01.698208 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:01.698209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:01.698219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:01.698259 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:01.698462 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.699045 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.701383 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.701900 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.701950 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.701960 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:01.702157 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:52:01.702201 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:52:01.702343 2864 raft_consensus.cc:2454] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbef088393c541b5a1ee29e8a6cf0f23 in term 0.
I20260619 02:52:01.702514 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.697186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:37829 (local address 127.21.164.193:56409)
0619 02:52:01.698070 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:01.698077 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:01.698141 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:01.698323 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:01.698324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:01.698326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:01.698393 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:01.698394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.699607 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.699608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:01.700846 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:01.700849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.702014 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.702014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:01.702082 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.702085 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:52:01.702397 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 02:52:01.702438 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":23,"thread_start_us":84,"threads_started":1}
I20260619 02:52:01.702577 2530 leader_election.cc:302] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f0662f15e964f078578118087757d37, bbef088393c541b5a1ee29e8a6cf0f23; no voters:
I20260619 02:52:01.702710 2951 raft_consensus.cc:2781] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:01.702759 2951 raft_consensus.cc:482] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:01.702785 2951 raft_consensus.cc:3037] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:01.702842 2730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703920f8492746b0bb53a583c6a80fdb" candidate_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" is_pre_election: true
I20260619 02:52:01.702996 2730 raft_consensus.cc:2454] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbef088393c541b5a1ee29e8a6cf0f23 in term 0.
I20260619 02:52:01.703671 2951 raft_consensus.cc:504] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0662f15e964f078578118087757d37" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 36697 } } peers { permanent_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37829 } } peers { permanent_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38823 } }
I20260619 02:52:01.703810 2951 leader_election.cc:288] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [CANDIDATE]: Term 1 election: Requested vote from peers 2f0662f15e964f078578118087757d37 (127.21.164.195:36697), f4f0c4ee0b1a46f8bb8f22598de54eda (127.21.164.194:37829)
I20260619 02:52:01.703979 2730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703920f8492746b0bb53a583c6a80fdb" candidate_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda"
I20260619 02:52:01.704032 2864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "703920f8492746b0bb53a583c6a80fdb" candidate_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0662f15e964f078578118087757d37"
I20260619 02:52:01.704059 2730 raft_consensus.cc:3037] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:01.704102 2864 raft_consensus.cc:3037] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:01.704963 2864 raft_consensus.cc:2454] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbef088393c541b5a1ee29e8a6cf0f23 in term 1.
I20260619 02:52:01.704977 2730 raft_consensus.cc:2454] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbef088393c541b5a1ee29e8a6cf0f23 in term 1.
I20260619 02:52:01.705148 2531 leader_election.cc:302] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbef088393c541b5a1ee29e8a6cf0f23, f4f0c4ee0b1a46f8bb8f22598de54eda; no voters:
I20260619 02:52:01.705266 2951 raft_consensus.cc:2781] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:01.705435 2951 raft_consensus.cc:686] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [term 1 LEADER]: Becoming Leader. State: Replica: bbef088393c541b5a1ee29e8a6cf0f23, State: Running, Role: LEADER
I20260619 02:52:01.705565 2951 consensus_queue.cc:237] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0662f15e964f078578118087757d37" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 36697 } } peers { permanent_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37829 } } peers { permanent_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38823 } }
I20260619 02:52:01.706472 2444 catalog_manager.cc:5495] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbef088393c541b5a1ee29e8a6cf0f23 (127.21.164.193). New cstate: current_term: 1 leader_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f0662f15e964f078578118087757d37" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 36697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbef088393c541b5a1ee29e8a6cf0f23" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 38823 } health_report { overall_health: HEALTHY } } }
I20260619 02:52:01.713708 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.710142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.21.164.193:38823)
0619 02:52:01.710181 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:01.710182 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:01.710188 (+ 6us) server_negotiation.cc:407] Connection header received
0619 02:52:01.710222 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:01.710223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:01.710254 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:01.710313 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:01.710757 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.711530 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.712668 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:01.712968 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:01.713020 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:01.713197 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:01.713342 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:01.713407 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:01.713604 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:52:01.713623 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:52:01.715801 2730 raft_consensus.cc:1261] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda [term 1 FOLLOWER]: Refusing update from remote peer bbef088393c541b5a1ee29e8a6cf0f23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:01.716282 2951 consensus_queue.cc:1035] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4f0c4ee0b1a46f8bb8f22598de54eda" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:01.718850 2864 raft_consensus.cc:1261] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37 [term 1 FOLLOWER]: Refusing update from remote peer bbef088393c541b5a1ee29e8a6cf0f23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:01.719195 2951 consensus_queue.cc:1035] T 703920f8492746b0bb53a583c6a80fdb P bbef088393c541b5a1ee29e8a6cf0f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f0662f15e964f078578118087757d37" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 36697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:01.722069 2959 mvcc.cc:205] Tried to move back new op lower bound from 7298406488945184768 to 7298406488906657792. Current Snapshot: MvccSnapshot[applied={T|T < 7298406488945184768}]
I20260619 02:52:01.723313 2960 mvcc.cc:205] Tried to move back new op lower bound from 7298406488945184768 to 7298406488906657792. Current Snapshot: MvccSnapshot[applied={T|T < 7298406488945184768}]
I20260619 02:52:01.727187 2443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:01.727306 2443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:01.728214 2443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82013c2f2d7343cebc4e913b4b80c526: Sending DeleteTablet for 3 replicas of tablet 703920f8492746b0bb53a583c6a80fdb
I20260619 02:52:01.728675 2695 tablet_service.cc:1514] Processing DeleteTablet for tablet 703920f8492746b0bb53a583c6a80fdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:01 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:48416
I20260619 02:52:01.729086 2576 tablet_service.cc:1514] Processing DeleteTablet for tablet 703920f8492746b0bb53a583c6a80fdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:01 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:60102
I20260619 02:52:01.729384 2844 tablet_service.cc:1514] Processing DeleteTablet for tablet 703920f8492746b0bb53a583c6a80fdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:01 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:35290
I20260619 02:52:01.730041 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 2482
I20260619 02:52:01.731798 2969 tablet_replica.cc:330] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37: stopping tablet replica
I20260619 02:52:01.731984 2969 raft_consensus.cc:2229] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:01.732159 2969 raft_consensus.cc:2258] T 703920f8492746b0bb53a583c6a80fdb P 2f0662f15e964f078578118087757d37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:01.732376 2967 tablet_replica.cc:330] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda: stopping tablet replica
I20260619 02:52:01.732527 2967 raft_consensus.cc:2229] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:01.732697 2967 raft_consensus.cc:2258] T 703920f8492746b0bb53a583c6a80fdb P f4f0c4ee0b1a46f8bb8f22598de54eda [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:52:01.736862 2431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:38823 (error 108)
I20260619 02:52:01.737751 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.737629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:38823 (local address 127.0.0.1:60108)
0619 02:52:01.737679 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:01.737712 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:38823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 02:52:01.737994 2431 catalog_manager.cc:4584] TS bbef088393c541b5a1ee29e8a6cf0f23 (127.21.164.193:38823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 703920f8492746b0bb53a583c6a80fdb: Network error: Client connection negotiation failed: client connection to 127.21.164.193:38823: connect: Connection refused (error 111)
I20260619 02:52:01.738279 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 2646
W20260619 02:52:01.745635 2937 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:52:01.745263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:37829 (local address 127.0.0.1:48420)
0619 02:52:01.745311 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:01.745321 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:01.745363 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:01.745572 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.194:37829: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 02:52:01.746155 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 2779
I20260619 02:52:01.764779 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:01.764622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:36697 (local address 127.0.0.1:35296)
0619 02:52:01.764701 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:01.764729 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.195:36697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 02:52:01.772946 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 2411
I20260619 02:52:01.778309 22163 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:52:02.123732 22163 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:52:02.123845 22163 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:52:02.458542 22163 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:52:02.458 UTC [1478] LOG: received smart shutdown request
2026-06-19 02:52:02.462 UTC [1478] LOG: background worker "logical replication launcher" (PID 1487) exited with exit code 1
2026-06-19 02:52:02.462 UTC [1482] LOG: shutting down
2026-06-19 02:52:02.623 UTC [1478] LOG: database system is shut down
2026-06-19T02:52:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47748 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:52:02 dist-test-slave-jt0t krb5kdc[2980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:52:02 dist-test-slave-jt0t krb5kdc[2980](info): set up 2 sockets
Jun 19 02:52:02 dist-test-slave-jt0t krb5kdc[2980](info): commencing operation
krb5kdc: starting...
W20260619 02:52:03.660377 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:52:03 dist-test-slave-jt0t krb5kdc[2980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:52:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:52:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:52:03.772241 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:33745
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:33745,127.21.164.253:38697,127.21.164.252:42435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:03.853298 2993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:03.853520 2993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:03.853543 2993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:03.854651 2993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:03.854683 2993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:03.854696 2993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:03.854707 2993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:03.854717 2993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:03.856133 2993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:33745,127.21.164.253:38697,127.21.164.252:42435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:33745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:03.856354 2993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:03.856550 2993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:03.861183 3001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:03.861406 2993 server_base.cc:1034] running on GCE node
W20260619 02:52:03.861207 2999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:03.861191 2998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:03.861804 2993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:03.862084 2993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:03.863243 2993 hybrid_clock.cc:648] HybridClock initialized: now 1781837523863226 us; error 31 us; skew 500 ppm
Jun 19 02:52:03 dist-test-slave-jt0t krb5kdc[2980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:03.870700 2993 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:52:03.871205 2993 webserver.cc:460] Webserver started at http://127.21.164.254:41923/ using document root <none> and password file <none>
I20260619 02:52:03.871421 2993 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:03.871470 2993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:03.871601 2993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:03.872736 2993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "1edc84eb87c443bfb91de84364e7b27d"
format_stamp: "Formatted at 2026-06-19 02:52:03 on dist-test-slave-jt0t"
server_key: "317b4ded563d2513313131456a4a7284"
server_key_iv: "0cf185709d511158e522323b5e72d4d8"
server_key_version: "encryptionkey@0"
I20260619 02:52:03.873070 2993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "1edc84eb87c443bfb91de84364e7b27d"
format_stamp: "Formatted at 2026-06-19 02:52:03 on dist-test-slave-jt0t"
server_key: "317b4ded563d2513313131456a4a7284"
server_key_iv: "0cf185709d511158e522323b5e72d4d8"
server_key_version: "encryptionkey@0"
I20260619 02:52:03.874327 2993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:03.875061 3008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:03.875218 2993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:52:03.875299 2993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "1edc84eb87c443bfb91de84364e7b27d"
format_stamp: "Formatted at 2026-06-19 02:52:03 on dist-test-slave-jt0t"
server_key: "317b4ded563d2513313131456a4a7284"
server_key_iv: "0cf185709d511158e522323b5e72d4d8"
server_key_version: "encryptionkey@0"
I20260619 02:52:03.875360 2993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:03.893385 2993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:03.895327 2993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:03.895478 2993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:03.899370 2993 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:33745
I20260619 02:52:03.899403 3060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:33745 every 8 connection(s)
I20260619 02:52:03.899726 2993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:03.900529 3061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:03.902117 3061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:52:03.907127 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 2993
I20260619 02:52:03.907238 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:52:03.907415 22163 external_mini_cluster.cc:1428] Setting key 1b5167c77c170f391b1b1b6f406058ae
Jun 19 02:52:03 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:03.911409 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:03.902812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:33745 (local address 127.0.0.1:50138)
0619 02:52:03.903044 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:03.903089 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:03.903622 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:03.903916 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:03.903918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:03.904192 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:03.904356 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:03.904360 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:03.905392 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:03.905394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:03.906060 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:03.906063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:03.906512 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:03.906513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:03.906577 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:03.907060 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:03.907069 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:03.908405 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:03.910062 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:03.910068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:03.910070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:03.910245 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:03.910444 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:03.910445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:03.910445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:03.910522 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:03.910862 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:03.910867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:03.910931 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 02:52:03.911010 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:52:03.911239 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":80,"thread_start_us":104,"threads_started":1}
I20260619 02:52:03.911412 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:03.902831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.21.164.254:33745)
0619 02:52:03.903055 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:03.903064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:03.903179 (+ 115us) server_negotiation.cc:407] Connection header received
0619 02:52:03.903785 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:03.903787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:03.903816 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:03.903897 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:03.904444 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:03.905309 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:03.906136 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:03.906437 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:03.906537 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:03.908482 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:03.908488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:03.908489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:03.908496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:03.909985 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:03.910313 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:03.910316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:03.910316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:03.910390 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:03.910573 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:03.910574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:03.910574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:03.910792 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:03.910872 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:03.911040 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 02:52:03.911156 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"spinlock_wait_cycles":8448,"thread_start_us":113,"threads_started":1}
I20260619 02:52:03.912209 3061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:52:03.912597 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:03.912461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:38697 (local address 127.0.0.1:59970)
0619 02:52:03.912513 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:03.912566 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 02:52:03.912748 3012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.253:38697: connect: Connection refused (error 111)
I20260619 02:52:03.913125 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:03.913031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:38697 (local address 127.0.0.1:59980)
0619 02:52:03.913074 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:03.913102 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 02:52:03.913225 3061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:38697: Network error: Client connection negotiation failed: client connection to 127.21.164.253:38697: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.21.164.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab.
Entry for principal kudu/127.21.164.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab.
I20260619 02:52:03.931181 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.253:38697
--webserver_interface=127.21.164.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab
--principal=kudu/127.21.164.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:33745,127.21.164.253:38697,127.21.164.252:42435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:03.962464 3061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } attempt: 1
I20260619 02:52:03.962991 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:03.962850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:38697 (local address 127.0.0.1:59986)
0619 02:52:03.962912 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:03.962959 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 02:52:03.963348 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:03.963265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:38697 (local address 127.0.0.1:59998)
0619 02:52:03.963306 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:03.963329 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:52:03.963456 3061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:38697: Network error: Client connection negotiation failed: client connection to 127.21.164.253:38697: connect: Connection refused (error 111)
W20260619 02:52:04.011392 3068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:04.011631 3068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:04.011658 3068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:04.012862 3068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:04.012903 3068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:04.012923 3068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:04.012938 3068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:04.012954 3068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:04.014457 3068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:33745,127.21.164.253:38697,127.21.164.252:42435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab
--principal=kudu/127.21.164.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.253:38697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.164.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:04.014699 3068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:04.014909 3068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:52:04.019672 3068 server_base.cc:1034] running on GCE node
W20260619 02:52:04.019656 3074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:04.019668 3073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:04.019640 3076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:04.020061 3068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:04.020385 3068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:04.021550 3068 hybrid_clock.cc:648] HybridClock initialized: now 1781837524021536 us; error 30 us; skew 500 ppm
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:04.024089 3068 init.cc:377] Logged in from keytab as kudu/127.21.164.253@KRBTEST.COM (short username kudu)
I20260619 02:52:04.024547 3068 webserver.cc:460] Webserver started at http://127.21.164.253:36881/ using document root <none> and password file <none>
I20260619 02:52:04.024763 3068 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:04.024811 3068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:04.024904 3068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:04.025925 3068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/data/instance:
uuid: "0a8caf25bf524613a13283072eda9050"
format_stamp: "Formatted at 2026-06-19 02:52:04 on dist-test-slave-jt0t"
server_key: "b508e21dde8c2adc48569a4e175ebe1e"
server_key_iv: "43120a4431a8514d7cbac4f2fa64578e"
server_key_version: "encryptionkey@0"
I20260619 02:52:04.026216 3068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/wal/instance:
uuid: "0a8caf25bf524613a13283072eda9050"
format_stamp: "Formatted at 2026-06-19 02:52:04 on dist-test-slave-jt0t"
server_key: "b508e21dde8c2adc48569a4e175ebe1e"
server_key_iv: "43120a4431a8514d7cbac4f2fa64578e"
server_key_version: "encryptionkey@0"
I20260619 02:52:04.027429 3068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:04.028136 3083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:04.028330 3068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:04.028419 3068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/wal
uuid: "0a8caf25bf524613a13283072eda9050"
format_stamp: "Formatted at 2026-06-19 02:52:04 on dist-test-slave-jt0t"
server_key: "b508e21dde8c2adc48569a4e175ebe1e"
server_key_iv: "43120a4431a8514d7cbac4f2fa64578e"
server_key_version: "encryptionkey@0"
I20260619 02:52:04.028488 3068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:04.031610 3061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } attempt: 2
I20260619 02:52:04.032007 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.031885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:38697 (local address 127.0.0.1:60012)
0619 02:52:04.031938 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.031975 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 02:52:04.032266 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.032212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:38697 (local address 127.0.0.1:60026)
0619 02:52:04.032231 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.032251 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260619 02:52:04.032341 3061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:38697: Network error: Client connection negotiation failed: client connection to 127.21.164.253:38697: connect: Connection refused (error 111)
I20260619 02:52:04.044513 3068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:04.046731 3068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:04.046849 3068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:04.050581 3068 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.253:38697
I20260619 02:52:04.050632 3135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.253:38697 every 8 connection(s)
I20260619 02:52:04.050956 3068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
I20260619 02:52:04.051803 3136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:04.053354 3136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:52:04.055847 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 3068
I20260619 02:52:04.055965 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-1/wal/instance
I20260619 02:52:04.056135 22163 external_mini_cluster.cc:1428] Setting key 9f22c837f4a600f6627cb0643d749434
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:04.062613 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.054078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.21.164.254:33745)
0619 02:52:04.054141 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.054144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.054322 (+ 178us) server_negotiation.cc:407] Connection header received
0619 02:52:04.054787 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.054789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.054801 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.054858 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.055484 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.056170 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.057216 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.057433 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.057491 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.059574 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.059576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.059577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.059585 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.061197 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.061626 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.061627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.061628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.061694 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.061921 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.061922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.061922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.062067 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.062133 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.062423 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.062508 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 02:52:04.062703 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.054037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:33745 (local address 127.0.0.1:50140)
0619 02:52:04.054200 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.054239 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:04.054688 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:04.054924 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:04.054927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:04.055173 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:04.055405 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.055409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.056291 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.056292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.057136 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.057138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.057505 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.057506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.057631 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.058108 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:04.058116 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:04.059485 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:04.061292 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.061300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.061303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.061546 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.061755 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.061756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.061756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.061867 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.062148 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:04.062155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:04.062294 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 02:52:04.062387 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:52:04.062519 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260619 02:52:04.063246 3136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.164.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.252@KRBTEST.COM" created.
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
I20260619 02:52:04.069986 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.063513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:38697 (local address 127.0.0.1:60040)
0619 02:52:04.063556 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.063566 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:04.063604 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:04.063858 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:04.063859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:04.063984 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:04.064031 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.064032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.064965 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.064966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.065474 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.065476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.066000 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.066000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.066060 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.066462 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:04.066467 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:04.067562 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:04.069012 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.069012 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.069013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.069177 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.069381 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.069382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.069382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.069437 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.069674 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:04.069675 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:04.069719 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:52:04.069779 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:52:04.069869 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 02:52:04.070005 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.063582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.21.164.253:38697)
0619 02:52:04.063726 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.063732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.063748 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:52:04.063767 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.063768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.063779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.063844 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.064093 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.064891 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.065540 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.065936 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.065997 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.067634 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.067642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.067642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.067648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.068938 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.069257 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.069260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.069261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.069326 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.069490 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.069491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.069491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.069624 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.069683 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.069815 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.069919 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260619 02:52:04.070760 3136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:52:04.071081 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.070963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43788)
0619 02:52:04.071008 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.071058 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:52:04.071221 3087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
I20260619 02:52:04.071656 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.071566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43792)
0619 02:52:04.071607 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.071635 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:52:04.071753 3136 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:42435: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab.
Entry for principal kudu/127.21.164.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab.
I20260619 02:52:04.081734 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.252:42435
--webserver_interface=127.21.164.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab
--principal=kudu/127.21.164.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:33745,127.21.164.253:38697,127.21.164.252:42435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:04.121027 3136 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } attempt: 1
I20260619 02:52:04.121608 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.121457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43808)
0619 02:52:04.121522 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.121571 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 02:52:04.121999 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.121912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43824)
0619 02:52:04.121951 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.121978 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 02:52:04.122108 3136 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:42435: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
I20260619 02:52:04.123497 3061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } attempt: 3
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
I20260619 02:52:04.130620 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.123792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:38697 (local address 127.0.0.1:60048)
0619 02:52:04.123846 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.123858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:04.123909 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:04.124065 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:04.124066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:04.124209 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:04.124275 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.124276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.125196 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.125197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.125711 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.125713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.126196 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.126197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.126299 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.126732 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:04.126737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:04.128082 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:04.129690 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.129691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.129692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.129856 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.130059 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.130060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.130060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.130127 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.130375 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:04.130375 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:04.130416 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 02:52:04.130465 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 02:52:04.130547 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 02:52:04.130661 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.123881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.21.164.253:38697)
0619 02:52:04.123932 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.123934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.123944 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:52:04.123971 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.123972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.123985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.124059 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.124337 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.125119 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.125783 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.126097 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.126166 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.128168 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.128170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.128170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.128178 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.129595 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.129916 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.129917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.129917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.129973 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.130188 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.130189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.130189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.130322 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.130381 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.130498 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.130572 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:52:04.131048 3061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:52:04.131402 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.131302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43830)
0619 02:52:04.131343 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.131381 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 02:52:04.131693 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.131628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43834)
0619 02:52:04.131661 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.131680 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 02:52:04.131774 3061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:42435: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
I20260619 02:52:04.162914 3061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } attempt: 1
I20260619 02:52:04.163470 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.163337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43842)
0619 02:52:04.163396 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.163439 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 02:52:04.163810 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.163706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43854)
0619 02:52:04.163756 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.163787 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 02:52:04.163905 3061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:42435: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
W20260619 02:52:04.163749 3143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:04.163978 3143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:04.164022 3143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:04.165257 3143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:04.165302 3143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:04.165323 3143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:04.165339 3143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:04.165364 3143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:04.166921 3143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:33745,127.21.164.253:38697,127.21.164.252:42435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab
--principal=kudu/127.21.164.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.252:42435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.164.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:04.167172 3143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:04.167385 3143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:04.172247 3151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:04.172312 3143 server_base.cc:1034] running on GCE node
W20260619 02:52:04.172276 3148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:04.172230 3149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:04.172618 3143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:04.172950 3143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:04.174093 3143 hybrid_clock.cc:648] HybridClock initialized: now 1781837524174069 us; error 36 us; skew 500 ppm
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:04.176541 3143 init.cc:377] Logged in from keytab as kudu/127.21.164.252@KRBTEST.COM (short username kudu)
I20260619 02:52:04.177047 3143 webserver.cc:460] Webserver started at http://127.21.164.252:44903/ using document root <none> and password file <none>
I20260619 02:52:04.177253 3143 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:04.177300 3143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:04.177428 3143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:04.178493 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/data/instance:
uuid: "ec163b5b8b034a4b9d5fc273432fa993"
format_stamp: "Formatted at 2026-06-19 02:52:04 on dist-test-slave-jt0t"
server_key: "0f62a5542c9a42504dec67fd41e20771"
server_key_iv: "08c880907927ab6c185d1e90060f83b2"
server_key_version: "encryptionkey@0"
I20260619 02:52:04.178819 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/wal/instance:
uuid: "ec163b5b8b034a4b9d5fc273432fa993"
format_stamp: "Formatted at 2026-06-19 02:52:04 on dist-test-slave-jt0t"
server_key: "0f62a5542c9a42504dec67fd41e20771"
server_key_iv: "08c880907927ab6c185d1e90060f83b2"
server_key_version: "encryptionkey@0"
I20260619 02:52:04.180049 3143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:52:04.180720 3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:04.180852 3143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:04.180936 3143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/wal
uuid: "ec163b5b8b034a4b9d5fc273432fa993"
format_stamp: "Formatted at 2026-06-19 02:52:04 on dist-test-slave-jt0t"
server_key: "0f62a5542c9a42504dec67fd41e20771"
server_key_iv: "08c880907927ab6c185d1e90060f83b2"
server_key_version: "encryptionkey@0"
I20260619 02:52:04.181002 3143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:04.190274 3136 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } attempt: 2
I20260619 02:52:04.190754 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.190605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43860)
0619 02:52:04.190669 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.190718 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 02:52:04.191109 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.191040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43862)
0619 02:52:04.191073 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.191094 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 02:52:04.191203 3136 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:42435: Network error: Client connection negotiation failed: client connection to 127.21.164.252:42435: connect: Connection refused (error 111)
I20260619 02:52:04.193715 3143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:04.195366 3143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:04.195492 3143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:04.199131 3143 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.252:42435
I20260619 02:52:04.199169 3210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.252:42435 every 8 connection(s)
I20260619 02:52:04.199493 3143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
I20260619 02:52:04.200313 3211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:04.201870 3211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:52:04.206229 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 3143
I20260619 02:52:04.206338 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837375929108-22163-0/minicluster-data/master-2/wal/instance
I20260619 02:52:04.206513 22163 external_mini_cluster.cc:1428] Setting key 25488f7e06b0687a67c64dd76bc82d5b
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:04.210132 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.202520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.21.164.254:33745)
0619 02:52:04.202583 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.202585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.202733 (+ 148us) server_negotiation.cc:407] Connection header received
0619 02:52:04.203164 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.203165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.203174 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.203229 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.203817 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.204275 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.205246 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.205472 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.205525 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.207518 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.207519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.207520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.207529 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.208889 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.209218 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.209218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.209219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.209276 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.209503 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.209504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.209504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.209619 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.209667 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.209961 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.210040 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:52:04.210265 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.202484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:33745 (local address 127.0.0.1:50142)
0619 02:52:04.202621 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.202663 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:04.203085 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:04.203291 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:04.203294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:04.203551 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:04.203745 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.203749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.204343 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.204344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.205173 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.205175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.205533 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.205534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.205633 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.206046 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:04.206054 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:04.207429 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:04.208954 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.208959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.208960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.209159 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.209361 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.209362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.209362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.209452 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.209668 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:04.209672 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:04.209824 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 02:52:04.209941 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:52:04.210078 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260619 02:52:04.211068 3211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
I20260619 02:52:04.214751 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.206868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.21.164.252:42435)
0619 02:52:04.207026 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.207030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.207049 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:52:04.207071 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.207073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.207092 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.207168 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.207820 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.208617 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.209534 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.209885 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.209944 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.212201 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.212209 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.212209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.212216 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.213718 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.214019 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.214022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.214023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.214071 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.214262 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.214263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.214263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.214387 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.214493 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.214590 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.214660 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260619 02:52:04.217399 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.211291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:38697 (local address 127.0.0.1:60052)
0619 02:52:04.211329 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.211339 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:04.211376 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:04.211505 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:04.211506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:04.211625 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:04.211666 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.211667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.212287 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.212288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.212951 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.212953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.213390 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.213391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.213440 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.213731 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:04.213735 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:04.215052 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:04.216513 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.216514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.216514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.216710 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.216909 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.216910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.216910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.216954 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.217172 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:04.217173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:04.217204 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:52:04.217245 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 02:52:04.217339 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 02:52:04.217438 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.211356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60052 (local address 127.21.164.253:38697)
0619 02:52:04.211414 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.211416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.211423 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:52:04.211445 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.211445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.211454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.211498 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.211727 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.212216 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.213030 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.213316 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.213394 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.215123 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.215124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.215125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.215133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.216442 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.216763 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.216764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.216765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.216847 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.216990 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.216991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.216991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.217118 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.217183 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.217270 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.217348 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:52:04.217774 3211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
I20260619 02:52:04.223125 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.215468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.21.164.254:33745)
0619 02:52:04.215509 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.215511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.215518 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:52:04.215537 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.215538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.215546 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.215590 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.216006 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.216446 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.217583 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.217843 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.217905 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.220442 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.220443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.220443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.220451 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.221885 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.222240 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.222240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.222241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.222294 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.222519 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.222521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.222521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.222669 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.222918 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.222960 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.223031 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:52:04.224682 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.218011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43870)
0619 02:52:04.218102 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.218111 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:04.218151 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:04.218259 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:04.218259 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:04.218372 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:04.218398 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.218399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.219253 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.219254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.219947 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.219950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.220312 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.220313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.220359 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.220733 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:04.220738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:04.222018 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:04.223804 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.223804 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.223805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.223983 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.224157 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.224158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.224159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.224228 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.224447 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:04.224447 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:04.224483 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:52:04.224530 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:52:04.224592 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 02:52:04.224812 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.218065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.21.164.252:42435)
0619 02:52:04.218098 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.218100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.218190 (+ 90us) server_negotiation.cc:407] Connection header received
0619 02:52:04.218202 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.218202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.218209 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.218242 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.218455 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.219157 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.220026 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.220244 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.220313 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.222133 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.222134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.222135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.222140 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.223723 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.224047 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.224048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.224049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.224100 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.224283 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.224284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.224284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.224389 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.224445 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.224567 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.224644 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:52:04.226111 3211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993: Bootstrap starting.
I20260619 02:52:04.226985 3211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:04.227257 3211 log.cc:824] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
I20260619 02:52:04.228034 3211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993: No bootstrap required, opened a new log
I20260619 02:52:04.229818 3211 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } }
I20260619 02:52:04.229969 3211 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:04.230002 3211 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec163b5b8b034a4b9d5fc273432fa993, State: Initialized, Role: FOLLOWER
I20260619 02:52:04.230154 3211 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } }
I20260619 02:52:04.230463 3211 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:04.230436 3217 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } } }
I20260619 02:52:04.230559 3217 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:52:04.230587 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.223741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60056 (local address 127.21.164.253:38697)
0619 02:52:04.223781 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.223782 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.223790 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:52:04.223823 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.223824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.223833 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.223871 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.224397 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.225094 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.225790 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.225986 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.226038 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.228195 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.228197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.228197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.228203 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.229463 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.229846 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.229848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.229848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.229904 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.230091 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.230092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.230093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.230212 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.230396 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.230431 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.230503 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:52:04.232311 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 02:52:04.232309 3228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:52:04.232432 3228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:52:04.232846 3143 master_runner.cc:186] Error getting master registration for 127.21.164.254:33745: OK, instance_id { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" instance_seqno: 1781837523898261 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:52:04.235056 3143 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:52:04.239027 3061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } attempt: 2
I20260619 02:52:04.240620 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.233880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.21.164.252:42435)
0619 02:52:04.233929 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.233931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.234579 (+ 648us) server_negotiation.cc:407] Connection header received
0619 02:52:04.234660 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.234661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.234670 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.234695 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.235328 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.235786 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.236669 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.236888 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.236917 (+ 29us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.238021 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.238022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.238023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.238029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.239391 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.239765 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.239766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.239766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.239822 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.240013 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.240014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.240014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.240180 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.240248 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.240451 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.240527 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 02:52:04.241671 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.233690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.21.164.253:38697)
0619 02:52:04.233729 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.233731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.233842 (+ 111us) server_negotiation.cc:407] Connection header received
0619 02:52:04.233933 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.233934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.233944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.233971 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.234508 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.235220 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.236320 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.236605 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.236702 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.238268 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.238270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.238270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.238278 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.239867 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.240230 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.240231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.240231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.240288 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.240481 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.240481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.240482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.240605 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.240719 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.241482 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.241566 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
I20260619 02:52:04.246183 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.239303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43896)
0619 02:52:04.239408 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.239418 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:04.239452 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:04.239826 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:04.239828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:04.239983 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:04.240034 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.240036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.240987 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.240989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.241657 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.241660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.242001 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.242002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.242053 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.242442 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:04.242447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:04.243597 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:04.245271 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.245272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.245273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.245436 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.245596 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.245597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.245598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.245653 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.245907 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:04.245908 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:04.245947 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 02:52:04.245990 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 02:52:04.246078 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 02:52:04.246587 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.239528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.21.164.252:42435)
0619 02:52:04.239689 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.239691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.239697 (+ 6us) server_negotiation.cc:407] Connection header received
0619 02:52:04.239713 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.239714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.239733 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.240094 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.240130 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.240888 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.241728 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.241925 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.242894 (+ 969us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.243671 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.243672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.243672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.243679 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.245210 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.245492 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.245493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.245493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.245545 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.245716 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.245717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.245717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.245836 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.246352 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.246392 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.246468 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260619 02:52:04.248131 3061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d: Bootstrap starting.
I20260619 02:52:04.248934 3061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:04.249229 3061 log.cc:824] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:04.250097 3061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d: No bootstrap required, opened a new log
I20260619 02:52:04.250864 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.244879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.21.164.254:33745)
0619 02:52:04.244927 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.244929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.244938 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:52:04.244984 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.244984 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.244994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.245053 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.245650 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.246325 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.247171 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.247369 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.247421 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.248466 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.248467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.248468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.248475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.249866 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.250181 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.250182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.250182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.250243 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.250408 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.250409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.250409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.250514 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.250593 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.250713 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.250782 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 02:52:04.252125 3061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } }
I20260619 02:52:04.252271 3061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:04.252303 3061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1edc84eb87c443bfb91de84364e7b27d, State: Initialized, Role: FOLLOWER
I20260619 02:52:04.252432 3061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } }
I20260619 02:52:04.252727 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } } }
I20260619 02:52:04.252779 3061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:04.252811 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:52:04.254549 3249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:52:04.254648 3249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:52:04.255427 2993 master_runner.cc:186] Error getting master registration for 127.21.164.253:38697: OK, instance_id { permanent_uuid: "0a8caf25bf524613a13283072eda9050" instance_seqno: 1781837524049502 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:52:04.257475 2993 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:52:04.282351 3136 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } attempt: 3
Jun 19 02:52:04 dist-test-slave-jt0t krb5kdc[2980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
I20260619 02:52:04.288744 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.282720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.21.164.252:42435)
0619 02:52:04.282758 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:04.282760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:04.282769 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:52:04.282789 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:04.282790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:04.282799 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:04.282845 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:04.283153 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.283647 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.284444 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.284678 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.284739 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.286432 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:04.286434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:04.286434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:04.286443 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:04.287688 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.287999 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.288000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.288001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.288053 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:04.288239 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:04.288240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:04.288240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:04.288352 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:04.288407 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:04.288542 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 02:52:04.288621 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:52:04.288744 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:04.282621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:42435 (local address 127.0.0.1:43902)
0619 02:52:04.282679 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:04.282692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:04.282736 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:04.282849 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:04.282850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:04.283030 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:04.283086 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.283088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.283719 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.283719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.284368 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:04.284371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:04.284737 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:04.284738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:04.284799 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:04.285212 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:04.285218 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:04.286357 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:04.287774 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.287775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.287775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.287938 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.288132 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:04.288132 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:04.288133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:04.288187 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:04.288407 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:04.288408 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:04.288456 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 02:52:04.288516 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:52:04.288619 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 02:52:04.290089 3136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050: Bootstrap starting.
I20260619 02:52:04.290702 3136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:04.290979 3136 log.cc:824] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:04.292115 3136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050: No bootstrap required, opened a new log
I20260619 02:52:04.293601 3136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } }
I20260619 02:52:04.293715 3136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:04.293749 3136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8caf25bf524613a13283072eda9050, State: Initialized, Role: FOLLOWER
I20260619 02:52:04.293885 3136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } }
I20260619 02:52:04.294129 3252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } } }
I20260619 02:52:04.294201 3136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:04.294263 3252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:52:04.295888 3263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:52:04.295991 3263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 02:52:04.297679 3089 tablet.cc:2369] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:04.298674 3068 master_runner.cc:263] No leader master found from master 0a8caf25bf524613a13283072eda9050
I20260619 02:52:04.298727 3068 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 02:52:04.396306 3014 tablet.cc:2369] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:04.410292 3252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:04.410432 3252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } }
I20260619 02:52:04.410696 3252 leader_election.cc:288] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1edc84eb87c443bfb91de84364e7b27d (127.21.164.254:33745), ec163b5b8b034a4b9d5fc273432fa993 (127.21.164.252:42435)
I20260619 02:52:04.411139 3185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a8caf25bf524613a13283072eda9050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec163b5b8b034a4b9d5fc273432fa993" is_pre_election: true
I20260619 02:52:04.411341 3185 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a8caf25bf524613a13283072eda9050 in term 0.
I20260619 02:52:04.411744 3035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a8caf25bf524613a13283072eda9050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1edc84eb87c443bfb91de84364e7b27d" is_pre_election: true
I20260619 02:52:04.411896 3035 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a8caf25bf524613a13283072eda9050 in term 0.
I20260619 02:52:04.412094 3087 leader_election.cc:302] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8caf25bf524613a13283072eda9050, ec163b5b8b034a4b9d5fc273432fa993; no voters:
I20260619 02:52:04.412245 3252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:04.412302 3252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:04.412330 3252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:04.413456 3252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } }
I20260619 02:52:04.413599 3252 leader_election.cc:288] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [CANDIDATE]: Term 1 election: Requested vote from peers 1edc84eb87c443bfb91de84364e7b27d (127.21.164.254:33745), ec163b5b8b034a4b9d5fc273432fa993 (127.21.164.252:42435)
I20260619 02:52:04.413955 3035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a8caf25bf524613a13283072eda9050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1edc84eb87c443bfb91de84364e7b27d"
I20260619 02:52:04.414049 3035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:04.414947 3035 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a8caf25bf524613a13283072eda9050 in term 1.
I20260619 02:52:04.415203 3185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a8caf25bf524613a13283072eda9050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec163b5b8b034a4b9d5fc273432fa993"
I20260619 02:52:04.415302 3185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:04.416132 3185 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a8caf25bf524613a13283072eda9050 in term 1.
I20260619 02:52:04.416435 3085 leader_election.cc:302] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8caf25bf524613a13283072eda9050, 1edc84eb87c443bfb91de84364e7b27d; no voters:
I20260619 02:52:04.416625 3252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:04.416787 3252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [term 1 LEADER]: Becoming Leader. State: Replica: 0a8caf25bf524613a13283072eda9050, State: Running, Role: LEADER
I20260619 02:52:04.416882 3252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } }
I20260619 02:52:04.417253 3252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a8caf25bf524613a13283072eda9050. Latest consensus state: current_term: 1 leader_uuid: "0a8caf25bf524613a13283072eda9050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } } }
I20260619 02:52:04.417342 3252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:04.418042 3267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:04.418206 3267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:04.419567 3185 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [term 1 FOLLOWER]: Refusing update from remote peer 0a8caf25bf524613a13283072eda9050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:04.419857 3265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:04.421074 3217 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a8caf25bf524613a13283072eda9050. Latest consensus state: current_term: 1 leader_uuid: "0a8caf25bf524613a13283072eda9050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } } }
I20260619 02:52:04.421166 3217 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:52:04.421535 3035 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [term 1 FOLLOWER]: Refusing update from remote peer 0a8caf25bf524613a13283072eda9050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:04.421774 3265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:04.422847 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a8caf25bf524613a13283072eda9050. Latest consensus state: current_term: 1 leader_uuid: "0a8caf25bf524613a13283072eda9050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } } }
I20260619 02:52:04.422932 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:52:04.423981 3252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a8caf25bf524613a13283072eda9050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } } }
I20260619 02:52:04.424062 3252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:04.424320 3238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a8caf25bf524613a13283072eda9050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } } }
I20260619 02:52:04.424407 3238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:52:04.425114 3267 catalog_manager.cc:1347] Generated new cluster ID: deabd7f3b3aa4eda88d38f18f94b4754
I20260619 02:52:04.425160 3267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:52:04.429704 3252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a8caf25bf524613a13283072eda9050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } } }
I20260619 02:52:04.429795 3252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:04.430203 3217 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a8caf25bf524613a13283072eda9050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edc84eb87c443bfb91de84364e7b27d" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 33745 } } peers { permanent_uuid: "0a8caf25bf524613a13283072eda9050" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 38697 } } peers { permanent_uuid: "ec163b5b8b034a4b9d5fc273432fa993" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 42435 } } }
I20260619 02:52:04.430321 3217 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:52:04.439329 3267 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:52:04.439990 3267 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:04.447880 3267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a8caf25bf524613a13283072eda9050: Generated new TSK 0
I20260619 02:52:04.448151 3267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:52:05.232951 3228 catalog_manager.cc:1259] Loaded cluster ID: deabd7f3b3aa4eda88d38f18f94b4754
I20260619 02:52:05.233031 3228 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993: loading cluster ID for follower catalog manager: success
I20260619 02:52:05.233850 3228 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993: acquiring CA information for follower catalog manager: success
I20260619 02:52:05.234126 3228 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec163b5b8b034a4b9d5fc273432fa993: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:52:05.255125 3249 catalog_manager.cc:1259] Loaded cluster ID: deabd7f3b3aa4eda88d38f18f94b4754
I20260619 02:52:05.255204 3249 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d: loading cluster ID for follower catalog manager: success
I20260619 02:52:05.256047 3249 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d: acquiring CA information for follower catalog manager: success
I20260619 02:52:05.256366 3249 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1edc84eb87c443bfb91de84364e7b27d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:52:05.513947 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 2993
I20260619 02:52:05.518396 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 3068
I20260619 02:52:05.522619 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 3143
2026-06-19T02:52:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2810 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:52:05 dist-test-slave-jt0t krb5kdc[3284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:52:05 dist-test-slave-jt0t krb5kdc[3284](info): set up 2 sockets
Jun 19 02:52:05 dist-test-slave-jt0t krb5kdc[3284](info): commencing operation
krb5kdc: starting...
W20260619 02:52:06.472391 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:52:06 dist-test-slave-jt0t krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:52:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:52:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:52:06.585760 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34169
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:33415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:34169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:06.665867 3297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:06.666085 3297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:06.666107 3297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:06.667268 3297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:06.667315 3297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:06.667328 3297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:06.667338 3297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:06.667349 3297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:06.668893 3297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:33415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:34169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:06.669134 3297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:06.669353 3297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:06.674141 3303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:06.674156 3302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:06.674201 3305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:06.674361 3297 server_base.cc:1034] running on GCE node
I20260619 02:52:06.674597 3297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:06.674919 3297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:06.676087 3297 hybrid_clock.cc:648] HybridClock initialized: now 1781837526676073 us; error 29 us; skew 500 ppm
Jun 19 02:52:06 dist-test-slave-jt0t krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837526, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:06.678707 3297 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:52:06.679190 3297 webserver.cc:460] Webserver started at http://127.21.164.254:34061/ using document root <none> and password file <none>
I20260619 02:52:06.679395 3297 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:06.679443 3297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:06.679565 3297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:06.680717 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "69214feb3a4d4f9898dfbff409932797"
format_stamp: "Formatted at 2026-06-19 02:52:06 on dist-test-slave-jt0t"
server_key: "113f55925d4bb2849a72363edfa6e98e"
server_key_iv: "e61443a64683f5a072dd1bab933219e7"
server_key_version: "encryptionkey@0"
I20260619 02:52:06.681079 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "69214feb3a4d4f9898dfbff409932797"
format_stamp: "Formatted at 2026-06-19 02:52:06 on dist-test-slave-jt0t"
server_key: "113f55925d4bb2849a72363edfa6e98e"
server_key_iv: "e61443a64683f5a072dd1bab933219e7"
server_key_version: "encryptionkey@0"
I20260619 02:52:06.682418 3297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:06.683210 3312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:06.683398 3297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:06.683482 3297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "69214feb3a4d4f9898dfbff409932797"
format_stamp: "Formatted at 2026-06-19 02:52:06 on dist-test-slave-jt0t"
server_key: "113f55925d4bb2849a72363edfa6e98e"
server_key_iv: "e61443a64683f5a072dd1bab933219e7"
server_key_version: "encryptionkey@0"
I20260619 02:52:06.683547 3297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:06.695230 3297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:06.697114 3297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:06.697249 3297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:06.701081 3297 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:34169
I20260619 02:52:06.701115 3364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:34169 every 8 connection(s)
I20260619 02:52:06.701452 3297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:06.702281 3365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:06.704533 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797: Bootstrap starting.
I20260619 02:52:06.705123 3365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:06.705348 3365 log.cc:824] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:06.705996 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797: No bootstrap required, opened a new log
I20260619 02:52:06.707517 3365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69214feb3a4d4f9898dfbff409932797" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34169 } }
I20260619 02:52:06.707661 3365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:06.707682 3365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69214feb3a4d4f9898dfbff409932797, State: Initialized, Role: FOLLOWER
I20260619 02:52:06.707790 3365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69214feb3a4d4f9898dfbff409932797" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34169 } }
I20260619 02:52:06.707840 3365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:06.707864 3365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:06.707891 3365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:06.708374 3365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69214feb3a4d4f9898dfbff409932797" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34169 } }
I20260619 02:52:06.708456 3365 leader_election.cc:302] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69214feb3a4d4f9898dfbff409932797; no voters:
I20260619 02:52:06.708577 3365 leader_election.cc:288] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:52:06.708633 3368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:06.708827 3368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [term 1 LEADER]: Becoming Leader. State: Replica: 69214feb3a4d4f9898dfbff409932797, State: Running, Role: LEADER
I20260619 02:52:06.708832 3365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:06.708976 3368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69214feb3a4d4f9898dfbff409932797" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34169 } }
I20260619 02:52:06.709371 3369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69214feb3a4d4f9898dfbff409932797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69214feb3a4d4f9898dfbff409932797" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34169 } } }
I20260619 02:52:06.709506 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69214feb3a4d4f9898dfbff409932797. Latest consensus state: current_term: 1 leader_uuid: "69214feb3a4d4f9898dfbff409932797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69214feb3a4d4f9898dfbff409932797" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34169 } } }
I20260619 02:52:06.709580 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:06.709508 3369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:06.710043 3377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:06.710330 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 3297
I20260619 02:52:06.710425 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:52:06.710606 22163 external_mini_cluster.cc:1428] Setting key 3b157fb8776198aeb0581c14f58cc3a4
I20260619 02:52:06.710606 3377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:06.712123 3377 catalog_manager.cc:1347] Generated new cluster ID: f285ae32d9534628adee27b11796d226
I20260619 02:52:06.712175 3377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:52:06 dist-test-slave-jt0t krb5kdc[3284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:06.720222 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:06.711203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.21.164.254:34169)
0619 02:52:06.711338 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:06.711348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:06.711379 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:52:06.711482 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:06.711488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:06.711534 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:06.711706 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:06.712397 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:06.713399 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:06.714119 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:06.714542 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:06.714635 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:06.716746 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:06.716750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:06.716751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:06.716763 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:06.718766 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:06.719073 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:06.719076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:06.719077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:06.719171 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:06.719341 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:06.719342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:06.719342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:06.719584 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:06.719718 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:06.719832 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 02:52:06.720054 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260619 02:52:06.721113 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:52:06.737731 3377 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:52:06.738250 3377 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:06.746593 3377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69214feb3a4d4f9898dfbff409932797: Generated new TSK 0
I20260619 02:52:06.746809 3377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:52:06.775027 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 3297
2026-06-19T02:52:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1251 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:52:06 dist-test-slave-jt0t krb5kdc[3392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:52:06 dist-test-slave-jt0t krb5kdc[3392](info): set up 2 sockets
Jun 19 02:52:06 dist-test-slave-jt0t krb5kdc[3392](info): commencing operation
krb5kdc: starting...
W20260619 02:52:07.720439 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:52:07 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:52:07.831447 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45997
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:45997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:07.920703 3405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:07.920934 3405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:07.920964 3405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:07.922122 3405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:07.922173 3405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:07.922192 3405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:07.922209 3405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:07.922223 3405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:07.923756 3405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:45997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:07.924007 3405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:07.924240 3405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:07.928849 3411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:07.928936 3413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:07.928877 3405 server_base.cc:1034] running on GCE node
W20260619 02:52:07.928865 3410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:07.929275 3405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:07.929546 3405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:07.930709 3405 hybrid_clock.cc:648] HybridClock initialized: now 1781837527930695 us; error 28 us; skew 500 ppm
Jun 19 02:52:07 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837527, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:07.933182 3405 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:52:07.933588 3405 webserver.cc:460] Webserver started at http://127.21.164.254:46833/ using document root <none> and password file <none>
I20260619 02:52:07.933797 3405 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:07.933846 3405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:07.933977 3405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:07.934991 3405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "650db6a9ae6a4e329a77f8588aee0309"
format_stamp: "Formatted at 2026-06-19 02:52:07 on dist-test-slave-jt0t"
server_key: "74fb30c2ce7aca8742deed4b81e019df"
server_key_iv: "58a0d3ea3619941981ab3b69e1d9200f"
server_key_version: "encryptionkey@0"
I20260619 02:52:07.935344 3405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "650db6a9ae6a4e329a77f8588aee0309"
format_stamp: "Formatted at 2026-06-19 02:52:07 on dist-test-slave-jt0t"
server_key: "74fb30c2ce7aca8742deed4b81e019df"
server_key_iv: "58a0d3ea3619941981ab3b69e1d9200f"
server_key_version: "encryptionkey@0"
I20260619 02:52:07.936553 3405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:07.937247 3420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:07.937390 3405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:07.937472 3405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "650db6a9ae6a4e329a77f8588aee0309"
format_stamp: "Formatted at 2026-06-19 02:52:07 on dist-test-slave-jt0t"
server_key: "74fb30c2ce7aca8742deed4b81e019df"
server_key_iv: "58a0d3ea3619941981ab3b69e1d9200f"
server_key_version: "encryptionkey@0"
I20260619 02:52:07.937530 3405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:07.972882 3405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:07.974819 3405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:07.974957 3405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:07.978760 3405 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45997
I20260619 02:52:07.978792 3472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45997 every 8 connection(s)
I20260619 02:52:07.979113 3405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:07.980073 3473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:07.982442 3473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Bootstrap starting.
I20260619 02:52:07.983000 3473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:07.983248 3473 log.cc:824] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:07.983959 3473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: No bootstrap required, opened a new log
I20260619 02:52:07.985591 3473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:07.985742 3473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:07.985764 3473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 650db6a9ae6a4e329a77f8588aee0309, State: Initialized, Role: FOLLOWER
I20260619 02:52:07.985909 3473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:07.985985 3473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:07.986008 3473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:07.986042 3473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:07.986124 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 3405
I20260619 02:52:07.986215 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:52:07.986380 22163 external_mini_cluster.cc:1428] Setting key 5ed11ae8e450e0ad68f4c761abca33f5
I20260619 02:52:07.987025 3473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:07.987138 3473 leader_election.cc:302] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 650db6a9ae6a4e329a77f8588aee0309; no voters:
I20260619 02:52:07.987280 3473 leader_election.cc:288] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:52:07.987303 3478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:07.987458 3478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 1 LEADER]: Becoming Leader. State: Replica: 650db6a9ae6a4e329a77f8588aee0309, State: Running, Role: LEADER
I20260619 02:52:07.987529 3473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:07.987547 3478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:07.987875 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 650db6a9ae6a4e329a77f8588aee0309. Latest consensus state: current_term: 1 leader_uuid: "650db6a9ae6a4e329a77f8588aee0309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } } }
I20260619 02:52:07.987943 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:07.988087 3479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "650db6a9ae6a4e329a77f8588aee0309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } } }
I20260619 02:52:07.988140 3479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:07.989364 3482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:07.989866 3482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:07.991688 3482 catalog_manager.cc:1347] Generated new cluster ID: 0fb934cda2644ad49adeba7a30389551
I20260619 02:52:07.991740 3482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:52:07 dist-test-slave-jt0t krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:07.996117 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:07.986867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.21.164.254:45997)
0619 02:52:07.987065 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:07.987075 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:07.987112 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:52:07.987246 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:07.987253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:07.987314 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:07.988091 (+ 777us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:07.988203 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:07.989283 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:07.990122 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:07.990567 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:07.990660 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:07.993257 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:07.993261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:07.993261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:07.993271 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:07.994766 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:07.995079 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:07.995082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:07.995082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:07.995176 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:07.995349 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:07.995350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:07.995351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:07.995579 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:07.995736 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:07.995805 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 02:52:07.995937 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260619 02:52:08.006362 3482 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:52:08.006927 3482 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:08.011758 3482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Generated new TSK 0
I20260619 02:52:08.011948 3482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:52:08.039577 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45997
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:08.139129 3499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:08.139349 3499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:08.139374 3499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:08.140574 3499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:08.140611 3499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:08.140640 3499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:52:08.142143 3499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:08.142382 3499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:08.142596 3499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:52:08.147418 3499 server_base.cc:1034] running on GCE node
W20260619 02:52:08.147400 3505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:08.147399 3504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:08.147395 3507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:08.147802 3499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:08.148078 3499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:08.149263 3499 hybrid_clock.cc:648] HybridClock initialized: now 1781837528149251 us; error 30 us; skew 500 ppm
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:08.151877 3499 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:52:08.152302 3499 webserver.cc:460] Webserver started at http://127.21.164.193:33743/ using document root <none> and password file <none>
I20260619 02:52:08.152501 3499 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:08.152542 3499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:08.152638 3499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:08.153795 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "33986f0294ab43f791372eef0cd62d73"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "bd2bd8ace69dfc8882822dc1384c2a4b"
server_key_iv: "f133393a042177b281752040c3f302d5"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.154083 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "33986f0294ab43f791372eef0cd62d73"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "bd2bd8ace69dfc8882822dc1384c2a4b"
server_key_iv: "f133393a042177b281752040c3f302d5"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.155340 3499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:52:08.156018 3514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.156169 3499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:52:08.156246 3499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "33986f0294ab43f791372eef0cd62d73"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "bd2bd8ace69dfc8882822dc1384c2a4b"
server_key_iv: "f133393a042177b281752040c3f302d5"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.156322 3499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:08.168821 3499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:08.171011 3499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:08.171156 3499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:08.171370 3499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:08.171681 3499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:08.171718 3499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.171741 3499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:08.171751 3499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.177294 3499 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:37891
I20260619 02:52:08.177359 3627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:37891 every 8 connection(s)
I20260619 02:52:08.177682 3499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:08.185346 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 3499
I20260619 02:52:08.185452 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:52:08.185622 22163 external_mini_cluster.cc:1428] Setting key 9701f286ccb7d6a2a8a807eb12660061
I20260619 02:52:08.186820 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.178396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:41577 (local address 127.21.164.254:45997)
0619 02:52:08.178443 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.178445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.178655 (+ 210us) server_negotiation.cc:407] Connection header received
0619 02:52:08.179176 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.179178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.179190 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.179244 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:08.180203 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.180740 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.181727 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.181931 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.181987 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.184022 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:08.184023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:08.184023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:08.184030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:08.185298 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.185720 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.185722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.185722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.185792 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.186065 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.186066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.186066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.186194 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:08.186256 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.186610 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.186727 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:52:08.187031 3630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.178361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:41577)
0619 02:52:08.178536 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.178573 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:08.179074 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:08.179336 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:08.179340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:08.179902 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:08.180102 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.180107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.180829 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.180830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.181639 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.181641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.182007 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.182008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.182132 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.182585 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:08.182593 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:08.183930 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:08.185375 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:08.185383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:08.185386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:08.185641 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:08.185869 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:08.185870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:08.185870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:08.186002 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:08.186350 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:08.186355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:08.186477 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 02:52:08.186571 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:52:08.186732 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260619 02:52:08.187462 3628 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45997
I20260619 02:52:08.187558 3628 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:08.187731 3628 heartbeater.cc:511] Master 127.21.164.254:45997 requested a full tablet report, sending...
I20260619 02:52:08.188254 3437 ts_manager.cc:194] Registered new tserver with Master: 33986f0294ab43f791372eef0cd62d73 (127.21.164.193:37891)
I20260619 02:52:08.188989 3437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:41577
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:52:08.208451 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45997
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:08.286832 3633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:08.287030 3633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:08.287052 3633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:08.288214 3633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:08.288267 3633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:08.288300 3633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:52:08.289855 3633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:08.290086 3633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:08.290324 3633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:08.295230 3638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:08.295202 3639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:08.295213 3633 server_base.cc:1034] running on GCE node
W20260619 02:52:08.295400 3641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:08.295619 3633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:08.295814 3633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:08.296943 3633 hybrid_clock.cc:648] HybridClock initialized: now 1781837528296933 us; error 29 us; skew 500 ppm
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:08.299214 3633 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:52:08.299649 3633 webserver.cc:460] Webserver started at http://127.21.164.194:38849/ using document root <none> and password file <none>
I20260619 02:52:08.299855 3633 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:08.299902 3633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:08.300022 3633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:08.301075 3633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "374ce4eb7d614bbb94ae42b1ef7e597a"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "cfdf242001f599b0abb15cea8f6cb5c1"
server_key_iv: "2ea12d3f96d59037c7052001d14574f5"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.301397 3633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "374ce4eb7d614bbb94ae42b1ef7e597a"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "cfdf242001f599b0abb15cea8f6cb5c1"
server_key_iv: "2ea12d3f96d59037c7052001d14574f5"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.302657 3633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:08.303396 3648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.303594 3633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:08.303687 3633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "374ce4eb7d614bbb94ae42b1ef7e597a"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "cfdf242001f599b0abb15cea8f6cb5c1"
server_key_iv: "2ea12d3f96d59037c7052001d14574f5"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.303752 3633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:08.320863 3633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:08.328771 3633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:08.328931 3633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:08.329169 3633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:08.329527 3633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:08.329572 3633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.329609 3633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:08.329633 3633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.335646 3633 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:40961
I20260619 02:52:08.335719 3761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:40961 every 8 connection(s)
I20260619 02:52:08.336014 3633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:08.343215 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 3633
I20260619 02:52:08.343333 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:52:08.343619 22163 external_mini_cluster.cc:1428] Setting key e5f50e0a2bdfb39a819b76c0a5469feb
I20260619 02:52:08.345551 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.336792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:38381 (local address 127.21.164.254:45997)
0619 02:52:08.336847 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.336849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.337059 (+ 210us) server_negotiation.cc:407] Connection header received
0619 02:52:08.337583 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.337584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.337598 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.337646 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:08.338649 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.339161 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.340196 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.340415 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.340471 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.342697 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:08.342699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:08.342699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:08.342707 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:08.344205 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.344620 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.344621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.344622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.344702 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.344924 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.344925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.344925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.345057 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:08.345119 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.345369 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.345459 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:52:08.345795 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.336767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:38381)
0619 02:52:08.336937 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.336976 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:08.337481 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:08.337741 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:08.337746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:08.338349 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:08.338552 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.338557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.339243 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.339244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.340121 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.340124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.340483 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.340484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.340620 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.341129 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:08.341137 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:08.342530 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:08.344285 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:08.344294 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:08.344297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:08.344544 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:08.344767 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:08.344767 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:08.344768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:08.344871 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:08.345110 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:08.345113 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:08.345235 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 02:52:08.345339 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 02:52:08.345467 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260619 02:52:08.346310 3762 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45997
I20260619 02:52:08.346411 3762 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:08.346611 3762 heartbeater.cc:511] Master 127.21.164.254:45997 requested a full tablet report, sending...
I20260619 02:52:08.346987 3437 ts_manager.cc:194] Registered new tserver with Master: 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961)
I20260619 02:52:08.347361 3437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:38381
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:52:08.367569 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45997
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:08.446806 3767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:08.447021 3767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:08.447052 3767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:08.448226 3767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:08.448271 3767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:08.448307 3767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:52:08.450004 3767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:08.450299 3767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:08.450524 3767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:08.455390 3775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:08.455385 3773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:08.455385 3772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:08.455523 3767 server_base.cc:1034] running on GCE node
I20260619 02:52:08.455770 3767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:08.456004 3767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:08.457139 3767 hybrid_clock.cc:648] HybridClock initialized: now 1781837528457111 us; error 49 us; skew 500 ppm
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:08.459592 3767 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:52:08.460048 3767 webserver.cc:460] Webserver started at http://127.21.164.195:46201/ using document root <none> and password file <none>
I20260619 02:52:08.460268 3767 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:08.460328 3767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:08.460455 3767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:08.461511 3767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "7b265ca155ad48ee817da237fdefc5e9"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "ddfbaa83e6f33271b6599af801404ecc"
server_key_iv: "beb95f613240faba2103ea9eb2f364cb"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.461844 3767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "7b265ca155ad48ee817da237fdefc5e9"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "ddfbaa83e6f33271b6599af801404ecc"
server_key_iv: "beb95f613240faba2103ea9eb2f364cb"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.463084 3767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:08.463759 3782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.463922 3767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:52:08.464001 3767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "7b265ca155ad48ee817da237fdefc5e9"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "ddfbaa83e6f33271b6599af801404ecc"
server_key_iv: "beb95f613240faba2103ea9eb2f364cb"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.464069 3767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:08.471441 3767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:08.473537 3767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:08.473671 3767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:08.473884 3767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:08.474169 3767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:08.474201 3767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.474231 3767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:08.474247 3767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.479658 3767 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:37487
I20260619 02:52:08.479698 3895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:37487 every 8 connection(s)
I20260619 02:52:08.480019 3767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:52:08.482048 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 3767
I20260619 02:52:08.482147 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:52:08.482308 22163 external_mini_cluster.cc:1428] Setting key f7d180a9ccd9185b9c73b0d22b6a64e6
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:08.489064 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.480734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:60679 (local address 127.21.164.254:45997)
0619 02:52:08.480800 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.480802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.480999 (+ 197us) server_negotiation.cc:407] Connection header received
0619 02:52:08.481541 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.481543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.481557 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.481607 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:08.482713 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.483381 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.484361 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.484543 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.484584 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.486541 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:08.486542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:08.486542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:08.486549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:08.487770 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.488127 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.488128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.488128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.488191 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.488450 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.488450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.488450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.488554 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:08.488633 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.488933 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.488997 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:52:08.489233 3898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.480719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:60679)
0619 02:52:08.480877 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.480913 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:08.481435 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:08.481679 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:08.481684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:08.482330 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:08.482617 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.482625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.483460 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.483461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.484293 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.484295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.484595 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.484595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.484704 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.485130 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:08.485138 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:08.486462 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:08.487833 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:08.487838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:08.487840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:08.488043 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:08.488294 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:08.488295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:08.488295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:08.488397 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:08.488625 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:08.488628 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:08.488825 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 02:52:08.488913 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:52:08.489036 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260619 02:52:08.489817 3896 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45997
I20260619 02:52:08.489904 3896 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:08.490065 3896 heartbeater.cc:511] Master 127.21.164.254:45997 requested a full tablet report, sending...
I20260619 02:52:08.490492 3437 ts_manager.cc:194] Registered new tserver with Master: 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487)
I20260619 02:52:08.490819 3437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:60679
I20260619 02:52:08.493834 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:52:08.500139 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.495093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.21.164.254:45997)
0619 02:52:08.495131 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.495133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.495232 (+ 99us) server_negotiation.cc:407] Connection header received
0619 02:52:08.495286 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.495286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.495295 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.495339 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:08.495820 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.496367 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.497064 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.497241 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.497289 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.498238 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:08.498239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:08.498239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:08.498245 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:08.499283 (+ 1038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.499565 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.499566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.499566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.499609 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.499720 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.499720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.499720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.499859 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:08.499945 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.500014 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.500074 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 02:52:08.501479 3437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:52:08.503064 3437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:52:08.509999 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.505542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:37891 (local address 127.0.0.1:44810)
0619 02:52:08.505682 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.505689 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:08.505722 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:08.506285 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:08.506286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:08.506288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:08.506350 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.506352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.507730 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.507731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.508795 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.508797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.509689 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.509690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.509786 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.509805 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:52:08.509854 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 02:52:08.509915 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":47,"threads_started":1}
I20260619 02:52:08.510115 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.505661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.21.164.195:37487)
0619 02:52:08.505825 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.505829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.505838 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:52:08.505869 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.505871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.505886 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.505960 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:08.506055 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.507194 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.508845 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.509587 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.509904 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.509970 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.509995 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.510038 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260619 02:52:08.510308 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.505331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:37487 (local address 127.0.0.1:39504)
0619 02:52:08.505626 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.505635 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:08.505673 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:08.505958 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:08.505959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:08.505960 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:08.505996 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.505997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.507307 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.507308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.508738 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.508741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.509687 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.509688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.509786 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.509805 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:52:08.509854 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 02:52:08.510228 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":326,"spinlock_wait_cycles":768,"thread_start_us":74,"threads_started":1}
I20260619 02:52:08.510504 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.505330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:40961 (local address 127.0.0.1:44346)
0619 02:52:08.505468 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.505512 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:08.505583 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:08.505884 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:08.505887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:08.505890 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:08.505957 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.505959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.507296 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.507297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.508732 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.508736 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.509689 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.509690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.509801 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.509806 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:52:08.509854 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:52:08.510425 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":530,"thread_start_us":77,"threads_started":1}
I20260619 02:52:08.510670 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.505928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.21.164.193:37891)
0619 02:52:08.506112 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.506117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.506125 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:52:08.506160 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.506162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.506176 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.506525 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:08.506597 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.507628 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.508879 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.509587 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.510465 (+ 878us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.510530 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.510554 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.510595 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":76,"threads_started":1}
I20260619 02:52:08.511073 3830 tablet_service.cc:1467] Processing CreateTablet for tablet 01aa16931cd640d681b59e08ab68e4a6 (DEFAULT_TABLE table=test-table [id=4127420a046b43cda05fde45ecfd262d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:08.511444 3830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01aa16931cd640d681b59e08ab68e4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:08.512068 3562 tablet_service.cc:1467] Processing CreateTablet for tablet 01aa16931cd640d681b59e08ab68e4a6 (DEFAULT_TABLE table=test-table [id=4127420a046b43cda05fde45ecfd262d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:08.512406 3562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01aa16931cd640d681b59e08ab68e4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:08.513435 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.505432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.21.164.194:40961)
0619 02:52:08.505622 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.505628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.505640 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:52:08.505680 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.505683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.505699 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.505790 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:08.506036 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.507176 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.508843 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.509587 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.509909 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.509975 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.510000 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.510044 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260619 02:52:08.514878 3910 tablet_bootstrap.cc:492] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9: Bootstrap starting.
I20260619 02:52:08.515444 3910 tablet_bootstrap.cc:654] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:08.515690 3911 tablet_bootstrap.cc:492] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Bootstrap starting.
I20260619 02:52:08.515970 3910 log.cc:824] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:08.516314 3911 tablet_bootstrap.cc:654] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:08.516451 3696 tablet_service.cc:1467] Processing CreateTablet for tablet 01aa16931cd640d681b59e08ab68e4a6 (DEFAULT_TABLE table=test-table [id=4127420a046b43cda05fde45ecfd262d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:08.516803 3696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01aa16931cd640d681b59e08ab68e4a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:08.516820 3911 log.cc:824] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:08.517532 3911 tablet_bootstrap.cc:492] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: No bootstrap required, opened a new log
I20260619 02:52:08.517619 3911 ts_tablet_manager.cc:1397] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:52:08.519038 3911 raft_consensus.cc:348] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:08.519176 3911 raft_consensus.cc:374] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:08.519207 3911 raft_consensus.cc:729] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33986f0294ab43f791372eef0cd62d73, State: Initialized, Role: FOLLOWER
I20260619 02:52:08.519325 3914 tablet_bootstrap.cc:492] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: Bootstrap starting.
I20260619 02:52:08.519341 3911 consensus_queue.cc:260] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:08.519614 3628 heartbeater.cc:503] Master 127.21.164.254:45997 was elected leader, sending a full tablet report...
I20260619 02:52:08.519645 3911 ts_tablet_manager.cc:1428] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:08.519934 3914 tablet_bootstrap.cc:654] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:08.520015 3910 tablet_bootstrap.cc:492] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9: No bootstrap required, opened a new log
I20260619 02:52:08.520092 3910 ts_tablet_manager.cc:1397] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 02:52:08.520509 3914 log.cc:824] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:08.521320 3914 tablet_bootstrap.cc:492] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: No bootstrap required, opened a new log
I20260619 02:52:08.521387 3914 ts_tablet_manager.cc:1397] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:52:08.522118 3910 raft_consensus.cc:348] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:08.522289 3910 raft_consensus.cc:374] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:08.522324 3910 raft_consensus.cc:729] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b265ca155ad48ee817da237fdefc5e9, State: Initialized, Role: FOLLOWER
I20260619 02:52:08.522439 3910 consensus_queue.cc:260] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:08.522724 3896 heartbeater.cc:503] Master 127.21.164.254:45997 was elected leader, sending a full tablet report...
I20260619 02:52:08.522710 3910 ts_tablet_manager.cc:1428] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:52:08.523216 3914 raft_consensus.cc:348] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:08.523382 3914 raft_consensus.cc:374] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:08.523412 3914 raft_consensus.cc:729] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374ce4eb7d614bbb94ae42b1ef7e597a, State: Initialized, Role: FOLLOWER
I20260619 02:52:08.523520 3914 consensus_queue.cc:260] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:08.523751 3914 ts_tablet_manager.cc:1428] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 02:52:08.523828 3762 heartbeater.cc:503] Master 127.21.164.254:45997 was elected leader, sending a full tablet report...
I20260619 02:52:08.566505 3915 raft_consensus.cc:482] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:08.566679 3915 raft_consensus.cc:504] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:08.567124 3915 leader_election.cc:288] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487), 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961)
I20260619 02:52:08.571020 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.567327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:60937 (local address 127.21.164.195:37487)
0619 02:52:08.567385 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.567387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.567396 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:52:08.567423 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.567424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.567432 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.567492 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:08.567643 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.568394 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.569911 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.570505 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.570584 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.570596 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.570899 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.570947 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:52:08.571020 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.567399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:44225 (local address 127.21.164.194:40961)
0619 02:52:08.567437 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.567439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.567518 (+ 79us) server_negotiation.cc:407] Connection header received
0619 02:52:08.567569 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.567570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.567581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.567623 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:08.567761 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.568429 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.569910 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.570497 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.570576 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.570586 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.570899 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.570946 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:52:08.571015 3630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.567241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:37487 (local address 127.21.164.193:60937)
0619 02:52:08.567274 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.567287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:08.567330 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:08.567501 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:08.567502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:08.567505 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:08.567582 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.567584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.568480 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.568481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.569805 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.569808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.570609 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.570610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.570795 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.570798 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:52:08.570858 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:52:08.570915 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":63}
I20260619 02:52:08.571023 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.567274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:40961 (local address 127.21.164.193:44225)
0619 02:52:08.567466 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.567475 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:08.567523 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:08.567642 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:08.567642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:08.567643 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:08.567699 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.567700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.568500 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.568501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.569805 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.569808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.570609 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.570610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.570790 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.570796 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:52:08.570858 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 02:52:08.570915 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":135,"thread_start_us":123,"threads_started":1}
I20260619 02:52:08.571381 3850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01aa16931cd640d681b59e08ab68e4a6" candidate_uuid: "33986f0294ab43f791372eef0cd62d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b265ca155ad48ee817da237fdefc5e9" is_pre_election: true
I20260619 02:52:08.571554 3850 raft_consensus.cc:2454] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33986f0294ab43f791372eef0cd62d73 in term 0.
I20260619 02:52:08.571565 3716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01aa16931cd640d681b59e08ab68e4a6" candidate_uuid: "33986f0294ab43f791372eef0cd62d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" is_pre_election: true
I20260619 02:52:08.571720 3716 raft_consensus.cc:2454] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33986f0294ab43f791372eef0cd62d73 in term 0.
I20260619 02:52:08.571802 3518 leader_election.cc:302] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33986f0294ab43f791372eef0cd62d73, 7b265ca155ad48ee817da237fdefc5e9; no voters:
I20260619 02:52:08.571936 3915 raft_consensus.cc:2781] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:08.571998 3915 raft_consensus.cc:482] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:08.572017 3915 raft_consensus.cc:3037] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:08.572739 3915 raft_consensus.cc:504] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:08.572866 3915 leader_election.cc:288] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 1 election: Requested vote from peers 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487), 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961)
I20260619 02:52:08.573035 3850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01aa16931cd640d681b59e08ab68e4a6" candidate_uuid: "33986f0294ab43f791372eef0cd62d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b265ca155ad48ee817da237fdefc5e9"
I20260619 02:52:08.573061 3716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01aa16931cd640d681b59e08ab68e4a6" candidate_uuid: "33986f0294ab43f791372eef0cd62d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a"
I20260619 02:52:08.573122 3850 raft_consensus.cc:3037] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:08.573137 3716 raft_consensus.cc:3037] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:08.574007 3850 raft_consensus.cc:2454] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33986f0294ab43f791372eef0cd62d73 in term 1.
I20260619 02:52:08.574007 3716 raft_consensus.cc:2454] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33986f0294ab43f791372eef0cd62d73 in term 1.
I20260619 02:52:08.574199 3518 leader_election.cc:302] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33986f0294ab43f791372eef0cd62d73, 7b265ca155ad48ee817da237fdefc5e9; no voters:
I20260619 02:52:08.574298 3915 raft_consensus.cc:2781] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:08.574461 3915 raft_consensus.cc:686] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 1 LEADER]: Becoming Leader. State: Replica: 33986f0294ab43f791372eef0cd62d73, State: Running, Role: LEADER
I20260619 02:52:08.574553 3915 consensus_queue.cc:237] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:08.575460 3437 catalog_manager.cc:5495] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33986f0294ab43f791372eef0cd62d73 (127.21.164.193). New cstate: current_term: 1 leader_uuid: "33986f0294ab43f791372eef0cd62d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } health_report { overall_health: HEALTHY } } }
I20260619 02:52:08.586229 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.583941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.21.164.193:37891)
0619 02:52:08.584019 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.584021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.584029 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:52:08.584063 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.584064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.584083 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.584156 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:08.584273 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.584797 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.585568 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.585768 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.585814 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.585957 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:08.586056 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:08.586113 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.586130 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.586149 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 02:52:08.586576 3763 tablet.cc:2369] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:08.588094 3850 raft_consensus.cc:1261] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9 [term 1 FOLLOWER]: Refusing update from remote peer 33986f0294ab43f791372eef0cd62d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:08.588106 3716 raft_consensus.cc:1261] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a [term 1 FOLLOWER]: Refusing update from remote peer 33986f0294ab43f791372eef0cd62d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:08.588455 3920 consensus_queue.cc:1035] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:08.588580 3915 consensus_queue.cc:1035] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:08.592850 3437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:08.592964 3437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:08.593670 3437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Sending DeleteTablet for 3 replicas of tablet 01aa16931cd640d681b59e08ab68e4a6
I20260619 02:52:08.594022 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 3499
I20260619 02:52:08.594192 3830 tablet_service.cc:1514] Processing DeleteTablet for tablet 01aa16931cd640d681b59e08ab68e4a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:08 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:39504
I20260619 02:52:08.594588 3696 tablet_service.cc:1514] Processing DeleteTablet for tablet 01aa16931cd640d681b59e08ab68e4a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:08 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:44346
I20260619 02:52:08.594796 3934 tablet_replica.cc:330] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: stopping tablet replica
I20260619 02:52:08.594903 3934 raft_consensus.cc:2229] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:08.595048 3934 raft_consensus.cc:2258] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:08.595429 3934 ts_tablet_manager.cc:1905] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:08.596813 3933 tablet_replica.cc:330] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9: stopping tablet replica
I20260619 02:52:08.596923 3934 ts_tablet_manager.cc:1918] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:08.596995 3934 log.cc:1196] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/01aa16931cd640d681b59e08ab68e4a6
I20260619 02:52:08.597059 3933 raft_consensus.cc:2229] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:08.597182 3933 raft_consensus.cc:2258] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:08.597317 3934 ts_tablet_manager.cc:1939] T 01aa16931cd640d681b59e08ab68e4a6 P 374ce4eb7d614bbb94ae42b1ef7e597a: Deleting consensus metadata
I20260619 02:52:08.597620 3933 ts_tablet_manager.cc:1905] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:08.597743 3422 catalog_manager.cc:4841] TS 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961): tablet 01aa16931cd640d681b59e08ab68e4a6 (table test-table [id=4127420a046b43cda05fde45ecfd262d]) successfully deleted
I20260619 02:52:08.598955 3933 ts_tablet_manager.cc:1918] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:08.599123 3933 log.cc:1196] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/01aa16931cd640d681b59e08ab68e4a6
W20260619 02:52:08.599212 3424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:37891 (error 108)
I20260619 02:52:08.599385 3933 ts_tablet_manager.cc:1939] T 01aa16931cd640d681b59e08ab68e4a6 P 7b265ca155ad48ee817da237fdefc5e9: Deleting consensus metadata
I20260619 02:52:08.599434 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 3633
I20260619 02:52:08.600013 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.599560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:37891 (local address 127.0.0.1:44822)
0619 02:52:08.599945 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.599981 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368}
W20260619 02:52:08.600129 3424 catalog_manager.cc:4584] TS 33986f0294ab43f791372eef0cd62d73 (127.21.164.193:37891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01aa16931cd640d681b59e08ab68e4a6: Network error: Client connection negotiation failed: client connection to 127.21.164.193:37891: connect: Connection refused (error 111)
I20260619 02:52:08.600541 3424 catalog_manager.cc:4841] TS 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487): tablet 01aa16931cd640d681b59e08ab68e4a6 (table test-table [id=4127420a046b43cda05fde45ecfd262d]) successfully deleted
I20260619 02:52:08.603721 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 3767
I20260619 02:52:08.607635 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 3405
I20260619 02:52:08.611986 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45997
--webserver_interface=127.21.164.254
--webserver_port=46833
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:45997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:08.692569 3936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:08.692808 3936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:08.692839 3936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:08.694062 3936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:08.694103 3936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:08.694121 3936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:08.694161 3936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:08.694191 3936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:08.695791 3936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:45997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:08.696046 3936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:08.696271 3936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:08.704608 3945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:08.704608 3943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:08.704732 3936 server_base.cc:1034] running on GCE node
W20260619 02:52:08.704603 3942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:08.705076 3936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:08.705387 3936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:08.706540 3936 hybrid_clock.cc:648] HybridClock initialized: now 1781837528706528 us; error 26 us; skew 500 ppm
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:08.709163 3936 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:52:08.709657 3936 webserver.cc:460] Webserver started at http://127.21.164.254:46833/ using document root <none> and password file <none>
I20260619 02:52:08.709892 3936 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:08.709952 3936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:08.711387 3936 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:08.712077 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.712299 3936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:08.712397 3936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "650db6a9ae6a4e329a77f8588aee0309"
format_stamp: "Formatted at 2026-06-19 02:52:07 on dist-test-slave-jt0t"
server_key: "74fb30c2ce7aca8742deed4b81e019df"
server_key_iv: "58a0d3ea3619941981ab3b69e1d9200f"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.712852 3936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:08.719230 3936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:08.721222 3936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:08.721342 3936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:08.725034 3936 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45997
I20260619 02:52:08.725064 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45997 every 8 connection(s)
I20260619 02:52:08.725433 3936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:08.726588 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 3936
I20260619 02:52:08.726892 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:37891
--local_ip_for_outbound_sockets=127.21.164.193
--tserver_master_addrs=127.21.164.254:45997
--webserver_port=33743
--webserver_interface=127.21.164.193
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:08.727286 4005 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:52:08.728133 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Bootstrap starting.
I20260619 02:52:08.736081 4005 log.cc:824] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:08.738070 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:08.738360 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Bootstrap complete.
I20260619 02:52:08.740438 4005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:08.740738 4005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 650db6a9ae6a4e329a77f8588aee0309, State: Initialized, Role: FOLLOWER
I20260619 02:52:08.740909 4005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:08.740967 4005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:08.741009 4005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:08.741055 4005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:52:08.742060 4005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:08.742151 4005 leader_election.cc:302] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 650db6a9ae6a4e329a77f8588aee0309; no voters:
I20260619 02:52:08.742322 4005 leader_election.cc:288] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:52:08.742373 4009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:52:08.742488 4009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 2 LEADER]: Becoming Leader. State: Replica: 650db6a9ae6a4e329a77f8588aee0309, State: Running, Role: LEADER
I20260619 02:52:08.742569 4009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:08.742575 4005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:08.742851 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "650db6a9ae6a4e329a77f8588aee0309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } } }
I20260619 02:52:08.742938 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:08.742857 4009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 650db6a9ae6a4e329a77f8588aee0309. Latest consensus state: current_term: 2 leader_uuid: "650db6a9ae6a4e329a77f8588aee0309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } } }
I20260619 02:52:08.743113 4009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:08.743423 4013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:08.744196 4013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:08.744302 4013 catalog_manager.cc:1259] Loaded cluster ID: 0fb934cda2644ad49adeba7a30389551
I20260619 02:52:08.744325 4013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:52:08.744546 4013 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 02:52:08.749130 4013 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 02:52:08.751744 4013 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:08.751950 4013 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Loaded TSK: 0
I20260619 02:52:08.752158 4013 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:52:08.809191 4007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:08.809399 4007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:08.809422 4007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:08.810606 4007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:08.810650 4007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:08.810678 4007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:52:08.812161 4007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:37891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=33743
--tserver_master_addrs=127.21.164.254:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:08.812381 4007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:08.812609 4007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:08.817530 4035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:08.817718 4007 server_base.cc:1034] running on GCE node
W20260619 02:52:08.817481 4033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:08.817481 4032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:08.817996 4007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:08.818279 4007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:08.819458 4007 hybrid_clock.cc:648] HybridClock initialized: now 1781837528819444 us; error 26 us; skew 500 ppm
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:08.822358 4007 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:52:08.822891 4007 webserver.cc:460] Webserver started at http://127.21.164.193:33743/ using document root <none> and password file <none>
I20260619 02:52:08.823151 4007 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:08.823228 4007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:08.824973 4007 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:08.825636 4042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.825800 4007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:52:08.825882 4007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "33986f0294ab43f791372eef0cd62d73"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "bd2bd8ace69dfc8882822dc1384c2a4b"
server_key_iv: "f133393a042177b281752040c3f302d5"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.826251 4007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:08.849875 4007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:08.852087 4007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:08.852229 4007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:08.852461 4007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:08.852953 4049 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:52:08.854445 4007 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:52:08.854497 4007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 02:52:08.854525 4007 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:52:08.855178 4007 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:52:08.855221 4007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 02:52:08.855281 4049 tablet_bootstrap.cc:492] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Bootstrap starting.
I20260619 02:52:08.862550 4007 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:37891
I20260619 02:52:08.863075 4007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:08.863674 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:37891 every 8 connection(s)
I20260619 02:52:08.865857 4049 log.cc:824] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:08.868330 4049 tablet_bootstrap.cc:492] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:08.868592 4049 tablet_bootstrap.cc:492] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Bootstrap complete.
I20260619 02:52:08.869477 4049 ts_tablet_manager.cc:1397] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260619 02:52:08.870725 4049 raft_consensus.cc:348] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:08.870973 4049 raft_consensus.cc:729] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33986f0294ab43f791372eef0cd62d73, State: Initialized, Role: FOLLOWER
I20260619 02:52:08.871102 4049 consensus_queue.cc:260] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:08.871356 4049 ts_tablet_manager.cc:1428] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:08.873373 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 4007
I20260619 02:52:08.873620 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:40961
--local_ip_for_outbound_sockets=127.21.164.194
--tserver_master_addrs=127.21.164.254:45997
--webserver_port=38849
--webserver_interface=127.21.164.194
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:08.881265 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.863862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:36535)
0619 02:52:08.864263 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.864311 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:08.864936 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:08.865567 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:08.865571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:08.865940 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:08.866184 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.866189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.867741 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.867742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.869092 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.869095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.869953 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.869955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.870100 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.875941 (+ 5841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:08.875951 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:08.877395 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:08.879556 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:08.879563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:08.879565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:08.879807 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:08.880103 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:08.880104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:08.880104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:08.880240 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:08.880748 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:08.880754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:08.880875 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 02:52:08.880952 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:52:08.881081 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":83,"threads_started":1}
I20260619 02:52:08.881409 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.864273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:36535 (local address 127.21.164.254:45997)
0619 02:52:08.865147 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.865156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.865189 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:52:08.865314 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.865319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.865370 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.865491 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:08.866311 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.867603 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.869220 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.869825 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.869939 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.877528 (+ 7589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:08.877534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:08.877535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:08.877550 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:08.879453 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.879900 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.879905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.879905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.880033 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.880311 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.880312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.880312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.880670 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:08.880763 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.881053 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.881206 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":110,"threads_started":1}
I20260619 02:52:08.882040 4157 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45997
I20260619 02:52:08.882143 4157 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:08.882321 4157 heartbeater.cc:511] Master 127.21.164.254:45997 requested a full tablet report, sending...
I20260619 02:52:08.883088 3969 ts_manager.cc:194] Registered new tserver with Master: 33986f0294ab43f791372eef0cd62d73 (127.21.164.193:37891)
I20260619 02:52:08.884338 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:36535
I20260619 02:52:08.885185 4157 heartbeater.cc:503] Master 127.21.164.254:45997 was elected leader, sending a full tablet report...
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:52:08.891683 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.883783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.21.164.193:37891)
0619 02:52:08.883945 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.883949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.884039 (+ 90us) server_negotiation.cc:407] Connection header received
0619 02:52:08.884126 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.884128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.884142 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.884198 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:08.884669 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.885804 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.886514 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.886996 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.887060 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.888870 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:08.888877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:08.888877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:08.888884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:08.890509 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.890841 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.890843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.890844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.890899 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:08.891043 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:08.891043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:08.891044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:08.891235 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:08.891308 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:08.891539 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 02:52:08.891617 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260619 02:52:08.891710 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.883706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:37891 (local address 127.0.0.1:44832)
0619 02:52:08.883942 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.883976 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:08.884045 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:08.884206 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:08.884210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:08.884514 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:08.884572 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.884575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.885874 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.885874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.886449 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.886451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.887057 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.887058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.887114 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.887565 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:08.887571 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:08.888798 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:08.890585 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:08.890591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:08.890593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:08.890776 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:08.890953 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:08.890953 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:08.890954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:08.891000 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:08.891339 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:08.891345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:08.891398 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 02:52:08.891475 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:52:08.891589 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
I20260619 02:52:08.892287 4077 tablet_service.cc:1514] Processing DeleteTablet for tablet 01aa16931cd640d681b59e08ab68e4a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:08 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:44832
I20260619 02:52:08.892833 4166 tablet_replica.cc:330] stopping tablet replica
I20260619 02:52:08.892920 4166 raft_consensus.cc:2229] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:08.892968 4166 raft_consensus.cc:2258] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:08.893239 4166 ts_tablet_manager.cc:1905] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:08.895221 4166 ts_tablet_manager.cc:1918] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:08.895303 4166 log.cc:1196] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal/wals/01aa16931cd640d681b59e08ab68e4a6
I20260619 02:52:08.895500 4166 ts_tablet_manager.cc:1939] T 01aa16931cd640d681b59e08ab68e4a6 P 33986f0294ab43f791372eef0cd62d73: Deleting consensus metadata
I20260619 02:52:08.895944 3956 catalog_manager.cc:4841] TS 33986f0294ab43f791372eef0cd62d73 (127.21.164.193:37891): tablet 01aa16931cd640d681b59e08ab68e4a6 (table test-table [id=4127420a046b43cda05fde45ecfd262d]) successfully deleted
W20260619 02:52:08.957912 4163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:08.958140 4163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:08.958174 4163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:08.959381 4163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:08.959430 4163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:08.959465 4163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:52:08.961030 4163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:40961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=38849
--tserver_master_addrs=127.21.164.254:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:08.961289 4163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:08.961513 4163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:08.966235 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:08.966354 4163 server_base.cc:1034] running on GCE node
W20260619 02:52:08.966251 4174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:08.966250 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:08.966660 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:08.966892 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:08.968052 4163 hybrid_clock.cc:648] HybridClock initialized: now 1781837528968038 us; error 32 us; skew 500 ppm
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:08.970571 4163 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:52:08.971007 4163 webserver.cc:460] Webserver started at http://127.21.164.194:38849/ using document root <none> and password file <none>
I20260619 02:52:08.971228 4163 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:08.971318 4163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:08.972673 4163 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:08.973223 4181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.973377 4163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.973453 4163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "374ce4eb7d614bbb94ae42b1ef7e597a"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "cfdf242001f599b0abb15cea8f6cb5c1"
server_key_iv: "2ea12d3f96d59037c7052001d14574f5"
server_key_version: "encryptionkey@0"
I20260619 02:52:08.973779 4163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:08.984608 4163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:08.986548 4163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:08.986663 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:08.986845 4163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:08.987126 4163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:08.987159 4163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.987183 4163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:08.987206 4163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:08.992777 4163 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:40961
I20260619 02:52:08.992820 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:40961 every 8 connection(s)
I20260619 02:52:08.993139 4163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:52:08.998158 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 4163
I20260619 02:52:08.998448 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:37487
--local_ip_for_outbound_sockets=127.21.164.195
--tserver_master_addrs=127.21.164.254:45997
--webserver_port=46201
--webserver_interface=127.21.164.195
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:52:08 dist-test-slave-jt0t krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:09.002085 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.993835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:32787 (local address 127.21.164.254:45997)
0619 02:52:08.993885 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:08.993887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:08.994051 (+ 164us) server_negotiation.cc:407] Connection header received
0619 02:52:08.994574 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:08.994576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:08.994592 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:08.994643 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:08.995645 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.996147 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.997120 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.997336 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.997392 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.999593 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:08.999595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:08.999595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:08.999603 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:09.000858 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.001206 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.001207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.001207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.001267 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.001479 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.001479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.001480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.001591 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:09.001646 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.001903 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.001993 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 02:52:09.002259 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:08.993776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:32787)
0619 02:52:08.993934 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:08.993974 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:08.994473 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:08.994744 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:08.994748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:08.995353 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:08.995546 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.995552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.996234 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.996235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.997038 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:08.997040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:08.997402 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:08.997403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:08.997518 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:08.997926 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:08.997934 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:08.999504 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:09.000928 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:09.000935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:09.000937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:09.001143 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:09.001320 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:09.001320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:09.001321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:09.001425 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:09.001645 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:09.001650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:09.001791 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 02:52:09.001877 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:52:09.002039 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260619 02:52:09.002782 4295 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45997
I20260619 02:52:09.002879 4295 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:09.003046 4295 heartbeater.cc:511] Master 127.21.164.254:45997 requested a full tablet report, sending...
I20260619 02:52:09.003430 3969 ts_manager.cc:194] Registered new tserver with Master: 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961)
I20260619 02:52:09.003932 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:32787
W20260619 02:52:09.079643 4298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:09.079859 4298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:09.079881 4298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:09.081043 4298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:09.081092 4298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:09.081118 4298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:52:09.082592 4298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:37487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=46201
--tserver_master_addrs=127.21.164.254:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:09.082892 4298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:09.083114 4298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:09.088107 4304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.088102 4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:09.088222 4298 server_base.cc:1034] running on GCE node
W20260619 02:52:09.088102 4303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:09.088567 4298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:09.088825 4298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:09.089968 4298 hybrid_clock.cc:648] HybridClock initialized: now 1781837529089957 us; error 25 us; skew 500 ppm
Jun 19 02:52:09 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:09.092397 4298 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:52:09.092841 4298 webserver.cc:460] Webserver started at http://127.21.164.195:46201/ using document root <none> and password file <none>
I20260619 02:52:09.093081 4298 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:09.093135 4298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:09.094439 4298 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:09.095054 4313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.095219 4298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:09.095314 4298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "7b265ca155ad48ee817da237fdefc5e9"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "ddfbaa83e6f33271b6599af801404ecc"
server_key_iv: "beb95f613240faba2103ea9eb2f364cb"
server_key_version: "encryptionkey@0"
I20260619 02:52:09.095651 4298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:09.119485 4298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:09.121639 4298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:09.121763 4298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:09.122004 4298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:09.122303 4298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:09.122336 4298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.122359 4298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:09.122383 4298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.127880 4298 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:37487
I20260619 02:52:09.127925 4426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:37487 every 8 connection(s)
I20260619 02:52:09.128248 4298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:52:09.133590 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 4298
Jun 19 02:52:09 dist-test-slave-jt0t krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:09.137132 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.128949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:39227 (local address 127.21.164.254:45997)
0619 02:52:09.129012 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:09.129015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:09.129254 (+ 239us) server_negotiation.cc:407] Connection header received
0619 02:52:09.129728 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:09.129729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:09.129741 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:09.129807 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:09.130729 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.131257 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.132197 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.132393 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.132457 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.134578 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:09.134579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:09.134580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:09.134587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:09.135846 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.136202 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.136203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.136203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.136258 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.136496 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.136497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.136497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.136605 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:09.136694 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.136958 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.137043 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:52:09.137313 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.128957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:39227)
0619 02:52:09.129099 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.129140 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:09.129636 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:09.129864 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:09.129867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:09.130429 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:09.130627 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.130632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.131330 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.131330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.132109 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.132111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.132462 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.132463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.132557 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.133030 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:09.133038 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:09.134501 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:09.135919 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:09.135926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:09.135928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:09.136138 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:09.136310 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:09.136310 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:09.136311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:09.136432 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:09.136695 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:09.136701 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:09.136830 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 02:52:09.136927 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 02:52:09.137081 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260619 02:52:09.137751 4427 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45997
I20260619 02:52:09.137851 4427 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:09.138049 4427 heartbeater.cc:511] Master 127.21.164.254:45997 requested a full tablet report, sending...
I20260619 02:52:09.138470 3969 ts_manager.cc:194] Registered new tserver with Master: 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487)
I20260619 02:52:09.138816 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:39227
I20260619 02:52:09.140056 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.133948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.21.164.254:45997)
0619 02:52:09.134057 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:09.134059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:09.134120 (+ 61us) server_negotiation.cc:407] Connection header received
0619 02:52:09.134182 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:09.134183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:09.134195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:09.134254 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:09.134740 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.135231 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.135920 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.136108 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.136161 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.137528 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:09.137529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:09.137530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:09.137537 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:09.138896 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.139333 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.139334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.139334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.139387 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:09.139591 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:09.139592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:09.139592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:09.139694 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:09.139849 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.139886 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.139963 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260619 02:52:09.140710 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:52:09.143391 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.141007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.21.164.254:45997)
0619 02:52:09.141042 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:09.141044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:09.141136 (+ 92us) server_negotiation.cc:407] Connection header received
0619 02:52:09.141206 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:09.141207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:09.141251 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:09.141305 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:09.141424 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.141891 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.142612 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.142909 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.142961 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.143111 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:09.143211 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:09.143287 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.143306 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.143328 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 02:52:09.143647 3969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:52:09.145313 3969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:52:09.147408 4077 tablet_service.cc:1467] Processing CreateTablet for tablet 2d05d4020a6a423983cd7c05f248c6be (DEFAULT_TABLE table=test-table [id=b8e36fa5fb504759baf661c15a3e3510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:09.147606 4077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d05d4020a6a423983cd7c05f248c6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:09.149153 4049 tablet_bootstrap.cc:492] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: Bootstrap starting.
I20260619 02:52:09.149462 4049 tablet_bootstrap.cc:654] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:09.149937 4049 tablet_bootstrap.cc:492] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: No bootstrap required, opened a new log
I20260619 02:52:09.149977 4049 ts_tablet_manager.cc:1397] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:52:09.150305 4049 raft_consensus.cc:348] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:09.150390 4049 raft_consensus.cc:374] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:09.150435 4049 raft_consensus.cc:729] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33986f0294ab43f791372eef0cd62d73, State: Initialized, Role: FOLLOWER
I20260619 02:52:09.150475 4049 consensus_queue.cc:260] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:09.150568 4049 ts_tablet_manager.cc:1428] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 02:52:09.150991 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.146981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:37487 (local address 127.0.0.1:39520)
0619 02:52:09.147081 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.147091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:09.147119 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:09.147456 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:09.147457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:09.147458 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:09.147500 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.147501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.148450 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.148451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.149509 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.149511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.150725 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.150726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.150800 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.150819 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:52:09.150873 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:52:09.150919 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 02:52:09.151003 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.147053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.21.164.195:37487)
0619 02:52:09.147313 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:09.147320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:09.147329 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:52:09.147366 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:09.147368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:09.147384 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:09.147455 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:09.147543 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.148371 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.149634 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.150650 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.150722 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.150781 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.150888 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.150925 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":64,"threads_started":1}
I20260619 02:52:09.151134 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.146999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:40961 (local address 127.0.0.1:44352)
0619 02:52:09.147296 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.147306 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:09.147358 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:09.147582 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:09.147583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:09.147586 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:09.147627 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.147629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.148933 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.148934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.149948 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.149950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.150894 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.150894 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.150957 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.150961 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:52:09.151008 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:52:09.151050 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":100,"threads_started":1}
I20260619 02:52:09.151347 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.147099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.21.164.194:40961)
0619 02:52:09.147301 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:09.147306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:09.147349 (+ 43us) server_negotiation.cc:407] Connection header received
0619 02:52:09.147417 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:09.147419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:09.147435 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:09.147513 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:09.147718 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.148833 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.150031 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.150741 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.150812 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.150865 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.151225 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.151270 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":108,"threads_started":1}
I20260619 02:52:09.151675 4361 tablet_service.cc:1467] Processing CreateTablet for tablet 2d05d4020a6a423983cd7c05f248c6be (DEFAULT_TABLE table=test-table [id=b8e36fa5fb504759baf661c15a3e3510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:09.151957 4361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d05d4020a6a423983cd7c05f248c6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:09.153340 4229 tablet_service.cc:1467] Processing CreateTablet for tablet 2d05d4020a6a423983cd7c05f248c6be (DEFAULT_TABLE table=test-table [id=b8e36fa5fb504759baf661c15a3e3510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:09.153613 4229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d05d4020a6a423983cd7c05f248c6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:09.154742 4437 tablet_bootstrap.cc:492] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9: Bootstrap starting.
I20260619 02:52:09.155308 4437 tablet_bootstrap.cc:654] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:09.155879 4437 log.cc:824] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:09.156284 4439 tablet_bootstrap.cc:492] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a: Bootstrap starting.
I20260619 02:52:09.156694 4437 tablet_bootstrap.cc:492] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9: No bootstrap required, opened a new log
I20260619 02:52:09.156765 4437 ts_tablet_manager.cc:1397] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:09.156934 4439 tablet_bootstrap.cc:654] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:09.157490 4439 log.cc:824] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:09.158281 4439 tablet_bootstrap.cc:492] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a: No bootstrap required, opened a new log
I20260619 02:52:09.158349 4439 ts_tablet_manager.cc:1397] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:09.158630 4437 raft_consensus.cc:348] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:09.158807 4437 raft_consensus.cc:374] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:09.158840 4437 raft_consensus.cc:729] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b265ca155ad48ee817da237fdefc5e9, State: Initialized, Role: FOLLOWER
I20260619 02:52:09.158958 4437 consensus_queue.cc:260] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:09.159235 4437 ts_tablet_manager.cc:1428] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:09.159350 4427 heartbeater.cc:503] Master 127.21.164.254:45997 was elected leader, sending a full tablet report...
I20260619 02:52:09.160249 4439 raft_consensus.cc:348] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:09.160377 4439 raft_consensus.cc:374] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:09.160396 4439 raft_consensus.cc:729] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374ce4eb7d614bbb94ae42b1ef7e597a, State: Initialized, Role: FOLLOWER
I20260619 02:52:09.160482 4439 consensus_queue.cc:260] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:09.160745 4439 ts_tablet_manager.cc:1428] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 02:52:09.160763 4295 heartbeater.cc:503] Master 127.21.164.254:45997 was elected leader, sending a full tablet report...
I20260619 02:52:09.164963 4162 raft_consensus.cc:482] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:09.165094 4162 raft_consensus.cc:504] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:09.165413 4162 leader_election.cc:288] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487), 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961)
I20260619 02:52:09.168007 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.165621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:46601 (local address 127.21.164.195:37487)
0619 02:52:09.165744 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:09.165746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:09.165754 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:52:09.165776 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:09.165776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:09.165784 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:09.165826 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:09.165963 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.166433 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.167311 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.167676 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.167743 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.167751 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.167903 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.167940 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 02:52:09.168008 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.165562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:37487 (local address 127.21.164.193:46601)
0619 02:52:09.165621 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.165632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:09.165681 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:09.165838 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:09.165838 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:09.165840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:09.165908 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.165910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.166500 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.166501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.167222 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.167223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.167742 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.167742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.167824 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.167845 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:52:09.167890 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 02:52:09.167934 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 02:52:09.168076 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.165621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:49121 (local address 127.21.164.194:40961)
0619 02:52:09.165667 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:09.165669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:09.165796 (+ 127us) server_negotiation.cc:407] Connection header received
0619 02:52:09.165848 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:09.165849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:09.165857 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:09.165908 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:09.166038 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.166493 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.167392 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.167769 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.167827 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.167835 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.167977 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.168012 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 02:52:09.168298 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d05d4020a6a423983cd7c05f248c6be" candidate_uuid: "33986f0294ab43f791372eef0cd62d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b265ca155ad48ee817da237fdefc5e9" is_pre_election: true
I20260619 02:52:09.168421 4381 raft_consensus.cc:2454] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33986f0294ab43f791372eef0cd62d73 in term 0.
I20260619 02:52:09.168468 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.165562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:40961 (local address 127.21.164.193:49121)
0619 02:52:09.165732 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.165740 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:09.165800 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:09.165914 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:09.165915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:09.165916 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:09.165987 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.165989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.166562 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.166563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.167315 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:09.167317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.167850 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.167851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:09.167920 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.167923 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:52:09.167966 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 02:52:09.168003 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":20,"thread_start_us":102,"threads_started":1}
I20260619 02:52:09.168638 4046 leader_election.cc:302] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33986f0294ab43f791372eef0cd62d73, 7b265ca155ad48ee817da237fdefc5e9; no voters:
I20260619 02:52:09.168797 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d05d4020a6a423983cd7c05f248c6be" candidate_uuid: "33986f0294ab43f791372eef0cd62d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" is_pre_election: true
I20260619 02:52:09.168924 4249 raft_consensus.cc:2454] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33986f0294ab43f791372eef0cd62d73 in term 0.
I20260619 02:52:09.168896 4162 raft_consensus.cc:2781] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:09.168960 4162 raft_consensus.cc:482] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:09.168977 4162 raft_consensus.cc:3037] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:09.169677 4162 raft_consensus.cc:504] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:09.169785 4162 leader_election.cc:288] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 1 election: Requested vote from peers 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487), 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961)
I20260619 02:52:09.169958 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d05d4020a6a423983cd7c05f248c6be" candidate_uuid: "33986f0294ab43f791372eef0cd62d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a"
I20260619 02:52:09.169945 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d05d4020a6a423983cd7c05f248c6be" candidate_uuid: "33986f0294ab43f791372eef0cd62d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b265ca155ad48ee817da237fdefc5e9"
I20260619 02:52:09.170027 4249 raft_consensus.cc:3037] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:09.170027 4381 raft_consensus.cc:3037] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:09.170897 4249 raft_consensus.cc:2454] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33986f0294ab43f791372eef0cd62d73 in term 1.
I20260619 02:52:09.170897 4381 raft_consensus.cc:2454] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33986f0294ab43f791372eef0cd62d73 in term 1.
I20260619 02:52:09.171068 4046 leader_election.cc:302] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33986f0294ab43f791372eef0cd62d73, 7b265ca155ad48ee817da237fdefc5e9; no voters:
I20260619 02:52:09.171193 4162 raft_consensus.cc:2781] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:09.171371 4162 raft_consensus.cc:686] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 LEADER]: Becoming Leader. State: Replica: 33986f0294ab43f791372eef0cd62d73, State: Running, Role: LEADER
I20260619 02:52:09.171437 4162 consensus_queue.cc:237] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:09.172233 3968 catalog_manager.cc:5495] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33986f0294ab43f791372eef0cd62d73 (127.21.164.193). New cstate: current_term: 1 leader_uuid: "33986f0294ab43f791372eef0cd62d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } health_report { overall_health: HEALTHY } } }
I20260619 02:52:09.179204 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.177142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.21.164.193:37891)
0619 02:52:09.177211 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:09.177212 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:09.177220 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:52:09.177259 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:09.177260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:09.177276 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:09.177357 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:09.177450 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.177920 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.178594 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:09.178812 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:09.178870 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:09.178965 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:09.179061 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:09.179109 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:09.179125 (+ 16us) server_negotiation.cc:299] Negotiation successful
0619 02:52:09.179149 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:52:09.180546 4249 raft_consensus.cc:1261] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a [term 1 FOLLOWER]: Refusing update from remote peer 33986f0294ab43f791372eef0cd62d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:09.180552 4381 raft_consensus.cc:1261] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9 [term 1 FOLLOWER]: Refusing update from remote peer 33986f0294ab43f791372eef0cd62d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:09.180867 4162 consensus_queue.cc:1035] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:09.180998 4444 consensus_queue.cc:1035] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:09.183429 4448 mvcc.cc:205] Tried to move back new op lower bound from 7298406519521316864 to 7298406519486922752. Current Snapshot: MvccSnapshot[applied={T|T < 7298406519521316864}]
I20260619 02:52:09.184363 4447 mvcc.cc:205] Tried to move back new op lower bound from 7298406519521316864 to 7298406519486922752. Current Snapshot: MvccSnapshot[applied={T|T < 7298406519521316864}]
I20260619 02:52:09.185199 3968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:09.185281 3968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:09.185918 3968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Sending DeleteTablet for 3 replicas of tablet 2d05d4020a6a423983cd7c05f248c6be
I20260619 02:52:09.186213 4077 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d05d4020a6a423983cd7c05f248c6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:09 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:44832
I20260619 02:52:09.186280 4229 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d05d4020a6a423983cd7c05f248c6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:09 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:44352
I20260619 02:52:09.186290 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d05d4020a6a423983cd7c05f248c6be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:09 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:39520
I20260619 02:52:09.186442 4166 tablet_replica.cc:330] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: stopping tablet replica
I20260619 02:52:09.186478 4455 tablet_replica.cc:330] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a: stopping tablet replica
I20260619 02:52:09.186504 4166 raft_consensus.cc:2229] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:52:09.186561 4455 raft_consensus.cc:2229] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:09.186579 4166 raft_consensus.cc:2258] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:09.186692 4455 raft_consensus.cc:2258] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:09.186688 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 4007
I20260619 02:52:09.187083 4455 ts_tablet_manager.cc:1905] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:09.188482 4455 ts_tablet_manager.cc:1918] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:09.188534 4455 log.cc:1196] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/2d05d4020a6a423983cd7c05f248c6be
I20260619 02:52:09.188803 4455 ts_tablet_manager.cc:1939] T 2d05d4020a6a423983cd7c05f248c6be P 374ce4eb7d614bbb94ae42b1ef7e597a: Deleting consensus metadata
I20260619 02:52:09.189209 3954 catalog_manager.cc:4841] TS 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961): tablet 2d05d4020a6a423983cd7c05f248c6be (table test-table [id=b8e36fa5fb504759baf661c15a3e3510]) successfully deleted
W20260619 02:52:09.191953 3956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:37891 (error 108)
I20260619 02:52:09.192162 4456 tablet_replica.cc:330] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9: stopping tablet replica
I20260619 02:52:09.192274 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 4163
I20260619 02:52:09.192287 4456 raft_consensus.cc:2229] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:09.192404 4456 raft_consensus.cc:2258] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:09.193305 4456 ts_tablet_manager.cc:1905] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:09.193560 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.193188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:37891 (local address 127.0.0.1:44854)
0619 02:52:09.193493 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.193529 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288}
W20260619 02:52:09.193672 3956 catalog_manager.cc:4584] TS 33986f0294ab43f791372eef0cd62d73 (127.21.164.193:37891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d05d4020a6a423983cd7c05f248c6be: Network error: Client connection negotiation failed: client connection to 127.21.164.193:37891: connect: Connection refused (error 111)
I20260619 02:52:09.194883 4456 ts_tablet_manager.cc:1918] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:09.194958 4456 log.cc:1196] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/2d05d4020a6a423983cd7c05f248c6be
I20260619 02:52:09.195221 4456 ts_tablet_manager.cc:1939] T 2d05d4020a6a423983cd7c05f248c6be P 7b265ca155ad48ee817da237fdefc5e9: Deleting consensus metadata
I20260619 02:52:09.195619 3956 catalog_manager.cc:4841] TS 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487): tablet 2d05d4020a6a423983cd7c05f248c6be (table test-table [id=b8e36fa5fb504759baf661c15a3e3510]) successfully deleted
I20260619 02:52:09.197470 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 4298
I20260619 02:52:09.201830 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 3936
I20260619 02:52:09.206110 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45997
--webserver_interface=127.21.164.254
--webserver_port=46833
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:45997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:09.284946 4458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:09.285166 4458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:09.285197 4458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:09.286365 4458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:09.286406 4458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:09.286424 4458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:09.286440 4458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:09.286455 4458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:09.287917 4458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:45997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:09.288156 4458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:09.288360 4458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:09.292960 4464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.292953 4463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.292956 4466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:09.293334 4458 server_base.cc:1034] running on GCE node
I20260619 02:52:09.293567 4458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:09.293843 4458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:09.294984 4458 hybrid_clock.cc:648] HybridClock initialized: now 1781837529294971 us; error 34 us; skew 500 ppm
Jun 19 02:52:09 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:09.297453 4458 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:52:09.297890 4458 webserver.cc:460] Webserver started at http://127.21.164.254:46833/ using document root <none> and password file <none>
I20260619 02:52:09.298103 4458 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:09.298152 4458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:09.299425 4458 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:09.299997 4473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.300148 4458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:52:09.300238 4458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "650db6a9ae6a4e329a77f8588aee0309"
format_stamp: "Formatted at 2026-06-19 02:52:07 on dist-test-slave-jt0t"
server_key: "74fb30c2ce7aca8742deed4b81e019df"
server_key_iv: "58a0d3ea3619941981ab3b69e1d9200f"
server_key_version: "encryptionkey@0"
I20260619 02:52:09.300580 4458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:09.332340 4458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:09.334510 4458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:09.334656 4458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:09.338460 4458 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45997
I20260619 02:52:09.338493 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45997 every 8 connection(s)
I20260619 02:52:09.338860 4458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:09.340432 4526 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:52:09.340863 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 4458
I20260619 02:52:09.341162 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:37891
--local_ip_for_outbound_sockets=127.21.164.193
--tserver_master_addrs=127.21.164.254:45997
--webserver_port=33743
--webserver_interface=127.21.164.193
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:09.341272 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Bootstrap starting.
I20260619 02:52:09.349499 4526 log.cc:824] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:09.351696 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:09.351985 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309: Bootstrap complete.
I20260619 02:52:09.354059 4526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:09.354297 4526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 650db6a9ae6a4e329a77f8588aee0309, State: Initialized, Role: FOLLOWER
I20260619 02:52:09.354470 4526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:09.354544 4526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:09.354593 4526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:09.354640 4526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:52:09.355610 4526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:09.355705 4526 leader_election.cc:302] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 650db6a9ae6a4e329a77f8588aee0309; no voters:
I20260619 02:52:09.355863 4526 leader_election.cc:288] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 02:52:09.355914 4530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:52:09.356021 4530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [term 3 LEADER]: Becoming Leader. State: Replica: 650db6a9ae6a4e329a77f8588aee0309, State: Running, Role: LEADER
I20260619 02:52:09.356105 4530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } }
I20260619 02:52:09.356133 4526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:09.356418 4531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "650db6a9ae6a4e329a77f8588aee0309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } } }
I20260619 02:52:09.356629 4531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:09.356580 4530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 650db6a9ae6a4e329a77f8588aee0309. Latest consensus state: current_term: 3 leader_uuid: "650db6a9ae6a4e329a77f8588aee0309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650db6a9ae6a4e329a77f8588aee0309" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45997 } } }
I20260619 02:52:09.357087 4540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:09.357267 4530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650db6a9ae6a4e329a77f8588aee0309 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:09.358151 4540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:09.358260 4540 catalog_manager.cc:1259] Loaded cluster ID: 0fb934cda2644ad49adeba7a30389551
I20260619 02:52:09.358291 4540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 02:52:09.358510 4540 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5c0a6a99ad google::LogMessage::Fail() at ??:0
@ 0x7f5c0a6ac88a google::LogMessage::SendToLog() at ??:0
@ 0x7f5c0a6a94de google::LogMessage::Flush() at ??:0
@ 0x7f5c0a6aaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5c125ca0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f5c125d38b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5c0c5770d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5c0c56c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5c0d3d06db start_thread at ??:0
@ 0x7f5c091a761f clone at ??:0
W20260619 02:52:09.432358 4528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:09.432564 4528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:09.432588 4528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:09.433777 4528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:09.433822 4528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:09.433849 4528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:52:09.435312 4528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:37891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=33743
--tserver_master_addrs=127.21.164.254:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:09.435527 4528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:09.435755 4528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:09.440512 4565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.440515 4563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.440538 4562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:09.440894 4528 server_base.cc:1034] running on GCE node
I20260619 02:52:09.441115 4528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:09.441372 4528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:09.442555 4528 hybrid_clock.cc:648] HybridClock initialized: now 1781837529442534 us; error 33 us; skew 500 ppm
Jun 19 02:52:09 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:09.445602 4528 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:52:09.446182 4528 webserver.cc:460] Webserver started at http://127.21.164.193:33743/ using document root <none> and password file <none>
I20260619 02:52:09.446440 4528 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:09.446511 4528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:09.448242 4528 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:52:09.449072 4572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.449251 4528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:52:09.449329 4528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "33986f0294ab43f791372eef0cd62d73"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "bd2bd8ace69dfc8882822dc1384c2a4b"
server_key_iv: "f133393a042177b281752040c3f302d5"
server_key_version: "encryptionkey@0"
I20260619 02:52:09.449676 4528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:09.472478 4528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:09.474644 4528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:09.474778 4528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:09.475019 4528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:09.475471 4579 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:52:09.477506 4528 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:52:09.477579 4528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 02:52:09.477620 4528 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:52:09.478281 4528 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:52:09.478312 4528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 02:52:09.478401 4579 tablet_bootstrap.cc:492] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: Bootstrap starting.
I20260619 02:52:09.484642 4528 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:37891
I20260619 02:52:09.484830 4686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:37891 every 8 connection(s)
I20260619 02:52:09.485086 4528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:09.486328 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 4528
I20260619 02:52:09.486629 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:40961
--local_ip_for_outbound_sockets=127.21.164.194
--tserver_master_addrs=127.21.164.254:45997
--webserver_port=38849
--webserver_interface=127.21.164.194
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:09.489466 4579 log.cc:824] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:09.492228 4579 tablet_bootstrap.cc:492] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:09.492588 4579 tablet_bootstrap.cc:492] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: Bootstrap complete.
I20260619 02:52:09.493701 4579 ts_tablet_manager.cc:1397] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20260619 02:52:09.495302 4579 raft_consensus.cc:348] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:09.495643 4579 raft_consensus.cc:729] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33986f0294ab43f791372eef0cd62d73, State: Initialized, Role: FOLLOWER
I20260619 02:52:09.495806 4579 consensus_queue.cc:260] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:09.496091 4579 ts_tablet_manager.cc:1428] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260619 02:52:09.538015 4689 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:52:09.485761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:49959)
0619 02:52:09.485967 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.486009 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:09.486634 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:09.537815 (+ 51181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
W20260619 02:52:09.538291 4687 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:45997 (0 consecutive failures): Network error: Failed to ping master at 127.21.164.254:45997: Client connection negotiation failed: client connection to 127.21.164.254:45997: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:52:09.538641 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.538530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:33697)
0619 02:52:09.538576 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.538617 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 02:52:09.539012 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.538899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:39221)
0619 02:52:09.538960 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.538990 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:52:09.539136 4687 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:52:09.570258 4690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:09.570472 4690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:09.570495 4690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:09.571692 4690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:09.571738 4690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:09.571763 4690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:52:09.573383 4690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:40961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=38849
--tserver_master_addrs=127.21.164.254:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:09.573643 4690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:09.573866 4690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:09.578701 4698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:09.578869 4690 server_base.cc:1034] running on GCE node
W20260619 02:52:09.578682 4700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.578682 4697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:09.579216 4690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:09.579450 4690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:09.580593 4690 hybrid_clock.cc:648] HybridClock initialized: now 1781837529580575 us; error 36 us; skew 500 ppm
Jun 19 02:52:09 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:09.582983 4690 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:52:09.583415 4690 webserver.cc:460] Webserver started at http://127.21.164.194:38849/ using document root <none> and password file <none>
I20260619 02:52:09.583631 4690 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:09.583680 4690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:09.584967 4690 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:09.585515 4707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.585652 4690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:52:09.585724 4690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "374ce4eb7d614bbb94ae42b1ef7e597a"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "cfdf242001f599b0abb15cea8f6cb5c1"
server_key_iv: "2ea12d3f96d59037c7052001d14574f5"
server_key_version: "encryptionkey@0"
I20260619 02:52:09.586061 4690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:09.592868 4690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:09.594919 4690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:09.595055 4690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:09.595302 4690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:09.595566 4690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:09.595595 4690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.595619 4690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:09.595638 4690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.601346 4690 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:40961
I20260619 02:52:09.601401 4820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:40961 every 8 connection(s)
I20260619 02:52:09.601753 4690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:52:09.602782 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.602375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:55597)
0619 02:52:09.602552 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.602635 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
W20260619 02:52:09.603226 4821 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:45997 (0 consecutive failures): Network error: Failed to ping master at 127.21.164.254:45997: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
I20260619 02:52:09.603507 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.603404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:37809)
0619 02:52:09.603457 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.603486 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 02:52:09.603741 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.603677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:42501)
0619 02:52:09.603706 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.603727 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260619 02:52:09.603820 4821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:52:09.611747 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 4690
I20260619 02:52:09.612006 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:37487
--local_ip_for_outbound_sockets=127.21.164.195
--tserver_master_addrs=127.21.164.254:45997
--webserver_port=46201
--webserver_interface=127.21.164.195
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:09.693048 4824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:09.693261 4824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:09.693287 4824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:09.694536 4824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:09.694583 4824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:09.694612 4824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:52:09.696151 4824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:37487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=46201
--tserver_master_addrs=127.21.164.254:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:09.696378 4824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:09.696609 4824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:09.701462 4832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.701485 4830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:09.701504 4829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:09.701548 4824 server_base.cc:1034] running on GCE node
I20260619 02:52:09.701819 4824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:09.702044 4824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:09.703178 4824 hybrid_clock.cc:648] HybridClock initialized: now 1781837529703170 us; error 29 us; skew 500 ppm
Jun 19 02:52:09 dist-test-slave-jt0t krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:09.705615 4824 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:52:09.706050 4824 webserver.cc:460] Webserver started at http://127.21.164.195:46201/ using document root <none> and password file <none>
I20260619 02:52:09.706254 4824 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:09.706300 4824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:09.707587 4824 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:09.708101 4839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.708258 4824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:52:09.708345 4824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "7b265ca155ad48ee817da237fdefc5e9"
format_stamp: "Formatted at 2026-06-19 02:52:08 on dist-test-slave-jt0t"
server_key: "ddfbaa83e6f33271b6599af801404ecc"
server_key_iv: "beb95f613240faba2103ea9eb2f364cb"
server_key_version: "encryptionkey@0"
I20260619 02:52:09.708724 4824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:09.717338 4824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:09.719199 4824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:09.719308 4824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:09.719488 4824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:09.719753 4824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:09.719781 4824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.719802 4824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:09.719818 4824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:09.725337 4824 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:37487
I20260619 02:52:09.725419 4952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:37487 every 8 connection(s)
I20260619 02:52:09.725708 4824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:52:09.726547 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 4824
I20260619 02:52:09.726825 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.726368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:50469)
0619 02:52:09.726528 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.726613 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
W20260619 02:52:09.727203 4953 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:45997 (0 consecutive failures): Network error: Failed to ping master at 127.21.164.254:45997: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
I20260619 02:52:09.727486 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.727381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:40573)
0619 02:52:09.727430 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.727464 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 02:52:09.727705 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:09.727655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:55019)
0619 02:52:09.727675 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:09.727692 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260619 02:52:09.727799 4953 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:52:10.540699 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:10.540462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:41863)
0619 02:52:10.540589 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:10.540620 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":52,"threads_started":1}
I20260619 02:52:10.604315 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:10.604101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:60339)
0619 02:52:10.604213 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:10.604265 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260619 02:52:10.728410 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:10.728160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:43853)
0619 02:52:10.728299 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:10.728353 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260619 02:52:11.047575 4960 raft_consensus.cc:482] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:11.047708 4960 raft_consensus.cc:504] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:11.048049 4960 leader_election.cc:288] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487), 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961)
Jun 19 02:52:11 dist-test-slave-jt0t krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.194@KRBTEST.COM
Jun 19 02:52:11 dist-test-slave-jt0t krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.195@KRBTEST.COM
I20260619 02:52:11.059870 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.048375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:33769 (local address 127.21.164.195:37487)
0619 02:52:11.048502 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:11.048515 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:11.048551 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:52:11.049153 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:11.049156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:11.049221 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:11.049378 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:11.050350 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.051774 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.053296 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.053957 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.054068 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:11.056965 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:11.056970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:11.056971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:11.056985 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:11.058569 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:11.058886 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:11.058888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:11.058889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:11.059003 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:11.059167 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:11.059168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:11.059168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:11.059377 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:11.059445 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:11.059638 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 02:52:11.059757 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260619 02:52:11.059865 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.048234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:40961 (local address 127.21.164.193:33489)
0619 02:52:11.048385 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:11.048395 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:11.048455 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:11.049436 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:11.049439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:11.049948 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:11.050214 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:11.050221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.051897 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.051897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:11.053215 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:11.053218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.054053 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.054053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:11.054208 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:11.054820 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:11.054831 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:11.056563 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:11.058338 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:11.058345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:11.058347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:11.058570 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:11.058801 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:11.058802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:11.058802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:11.058908 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:11.059338 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:11.059345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:11.059472 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 02:52:11.059562 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:52:11.059764 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":61,"thread_start_us":74,"threads_started":1}
I20260619 02:52:11.059875 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.048347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:33489 (local address 127.21.164.194:40961)
0619 02:52:11.048520 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:11.048527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:11.048551 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:52:11.049153 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:11.049156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:11.049240 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:11.049378 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:11.050350 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.051791 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.053296 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.053957 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.054068 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:11.056704 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:11.056708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:11.056709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:11.056719 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:11.058257 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:11.058645 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:11.058648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:11.058648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:11.058741 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:11.058963 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:11.058964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:11.058964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:11.059280 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:11.059348 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:11.059622 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 02:52:11.059753 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":54,"threads_started":1}
I20260619 02:52:11.059957 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.048237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:37487 (local address 127.21.164.193:33769)
0619 02:52:11.048480 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:11.048489 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:11.048525 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:11.049436 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:11.049439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:11.049948 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:11.050214 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:11.050221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.051884 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.051885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:11.053215 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:11.053218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:11.054052 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:11.054054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:11.054236 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:11.054837 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:11.054841 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:11.056861 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:11.058648 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:11.058649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:11.058650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:11.058813 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:11.059063 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:11.059064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:11.059064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:11.059117 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:11.059446 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:11.059448 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:11.059496 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 02:52:11.059566 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:52:11.059842 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":194,"thread_start_us":65,"threads_started":1}
W20260619 02:52:11.060925 4574 leader_election.cc:341] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961): Not found: Tablet not found: 2d05d4020a6a423983cd7c05f248c6be
W20260619 02:52:11.061460 4576 leader_election.cc:341] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487): Not found: Tablet not found: 2d05d4020a6a423983cd7c05f248c6be
I20260619 02:52:11.061510 4576 leader_election.cc:302] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33986f0294ab43f791372eef0cd62d73; no voters: 374ce4eb7d614bbb94ae42b1ef7e597a, 7b265ca155ad48ee817da237fdefc5e9
I20260619 02:52:11.061623 4960 raft_consensus.cc:2726] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:52:11.541215 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.541082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:52589)
0619 02:52:11.541138 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:11.541171 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 02:52:11.604936 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.604752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:60785)
0619 02:52:11.604860 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:11.604894 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260619 02:52:11.729123 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:11.728907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:33187)
0619 02:52:11.729036 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:11.729082 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260619 02:52:12.542035 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:12.541728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:53397)
0619 02:52:12.541926 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:12.541973 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":112,"threads_started":1}
I20260619 02:52:12.605666 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:12.605424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:53981)
0619 02:52:12.605581 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:12.605617 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260619 02:52:12.729848 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:12.729581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:53251)
0619 02:52:12.729741 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:12.729794 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260619 02:52:12.941313 4970 raft_consensus.cc:482] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:12.941431 4970 raft_consensus.cc:504] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:12.941630 4970 leader_election.cc:288] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487), 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961)
W20260619 02:52:12.942078 4574 leader_election.cc:341] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961): Not found: Tablet not found: 2d05d4020a6a423983cd7c05f248c6be
W20260619 02:52:12.942160 4576 leader_election.cc:341] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487): Not found: Tablet not found: 2d05d4020a6a423983cd7c05f248c6be
I20260619 02:52:12.942190 4576 leader_election.cc:302] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33986f0294ab43f791372eef0cd62d73; no voters: 374ce4eb7d614bbb94ae42b1ef7e597a, 7b265ca155ad48ee817da237fdefc5e9
I20260619 02:52:12.942312 4970 raft_consensus.cc:2726] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:52:13.542851 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:13.542576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:35615)
0619 02:52:13.542734 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:13.542781 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260619 02:52:13.606453 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:13.606191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:36089)
0619 02:52:13.606360 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:13.606395 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260619 02:52:13.730588 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:13.730334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:38379)
0619 02:52:13.730481 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:13.730528 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260619 02:52:14.543629 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.543404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:43141)
0619 02:52:14.543548 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.543582 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260619 02:52:14.607219 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.606981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:50391)
0619 02:52:14.607120 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.607170 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260619 02:52:14.731288 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:14.731039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:48665)
0619 02:52:14.731197 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:14.731243 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":62,"threads_started":1}
I20260619 02:52:15.009418 4977 raft_consensus.cc:482] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:15.009514 4977 raft_consensus.cc:504] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:15.009716 4977 leader_election.cc:288] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487), 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961)
W20260619 02:52:15.010120 4574 leader_election.cc:341] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961): Not found: Tablet not found: 2d05d4020a6a423983cd7c05f248c6be
W20260619 02:52:15.010216 4576 leader_election.cc:341] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487): Not found: Tablet not found: 2d05d4020a6a423983cd7c05f248c6be
I20260619 02:52:15.010280 4576 leader_election.cc:302] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33986f0294ab43f791372eef0cd62d73; no voters: 374ce4eb7d614bbb94ae42b1ef7e597a, 7b265ca155ad48ee817da237fdefc5e9
I20260619 02:52:15.010360 4977 raft_consensus.cc:2726] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:52:15.544332 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.544119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:55423)
0619 02:52:15.544241 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.544283 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260619 02:52:15.607810 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.607603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:33283)
0619 02:52:15.607719 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.607756 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260619 02:52:15.732080 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:15.731822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:55773)
0619 02:52:15.731989 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:15.732035 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":61,"threads_started":1}
I20260619 02:52:16.545042 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:16.544832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:53401)
0619 02:52:16.544949 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:16.544991 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260619 02:52:16.608553 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:16.608256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:52325)
0619 02:52:16.608441 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:16.608490 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260619 02:52:16.732872 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:16.732560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:52043)
0619 02:52:16.732759 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:16.732814 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":127,"threads_started":1}
I20260619 02:52:17.545758 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:17.545493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:51865)
0619 02:52:17.545654 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:17.545695 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260619 02:52:17.609217 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:17.608987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:46951)
0619 02:52:17.609123 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:17.609164 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260619 02:52:17.733621 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:17.733361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:54071)
0619 02:52:17.733526 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:17.733567 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
I20260619 02:52:18.546518 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:18.546240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:60531)
0619 02:52:18.546411 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:18.546452 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260619 02:52:18.609931 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:18.609666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:50173)
0619 02:52:18.609828 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:18.609872 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260619 02:52:18.734375 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:18.734120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:47399)
0619 02:52:18.734282 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:18.734323 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260619 02:52:18.834769 4990 raft_consensus.cc:482] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:18.834885 4990 raft_consensus.cc:504] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b265ca155ad48ee817da237fdefc5e9" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 37487 } } peers { permanent_uuid: "374ce4eb7d614bbb94ae42b1ef7e597a" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 40961 } } peers { permanent_uuid: "33986f0294ab43f791372eef0cd62d73" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 37891 } }
I20260619 02:52:18.835098 4990 leader_election.cc:288] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487), 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961)
W20260619 02:52:18.835438 4574 leader_election.cc:341] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 374ce4eb7d614bbb94ae42b1ef7e597a (127.21.164.194:40961): Not found: Tablet not found: 2d05d4020a6a423983cd7c05f248c6be
W20260619 02:52:18.835516 4576 leader_election.cc:341] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b265ca155ad48ee817da237fdefc5e9 (127.21.164.195:37487): Not found: Tablet not found: 2d05d4020a6a423983cd7c05f248c6be
I20260619 02:52:18.835556 4576 leader_election.cc:302] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33986f0294ab43f791372eef0cd62d73; no voters: 374ce4eb7d614bbb94ae42b1ef7e597a, 7b265ca155ad48ee817da237fdefc5e9
I20260619 02:52:18.835621 4990 raft_consensus.cc:2726] T 2d05d4020a6a423983cd7c05f248c6be P 33986f0294ab43f791372eef0cd62d73 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:52:19.547346 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.547048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:34933)
0619 02:52:19.547223 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:19.547280 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260619 02:52:19.610633 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.610409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:59335)
0619 02:52:19.610538 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:19.610581 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260619 02:52:19.735175 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:19.734906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:50259)
0619 02:52:19.735070 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:19.735113 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20260619 02:52:20.548240 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.547919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:39161)
0619 02:52:20.548100 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.548174 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260619 02:52:20.611470 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.611167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:52179)
0619 02:52:20.611325 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.611398 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260619 02:52:20.735944 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:20.735699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:49113)
0619 02:52:20.735846 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:20.735893 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260619 02:52:21.548990 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:21.548765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:33951)
0619 02:52:21.548905 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.548942 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260619 02:52:21.612177 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:21.611934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:54539)
0619 02:52:21.612085 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.612125 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260619 02:52:21.736702 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:21.736431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:34003)
0619 02:52:21.736568 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:21.736615 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260619 02:52:22.549674 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:22.549415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:44247)
0619 02:52:22.549567 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:22.549609 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
I20260619 02:52:22.612903 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:22.612641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:42785)
0619 02:52:22.612813 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:22.612852 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":108,"threads_started":1}
I20260619 02:52:22.737560 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:22.737264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:37341)
0619 02:52:22.737450 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:22.737497 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":118,"threads_started":1}
I20260619 02:52:23.550397 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:23.550140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:37415)
0619 02:52:23.550292 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:23.550333 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260619 02:52:23.613672 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:23.613399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:42045)
0619 02:52:23.613558 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:23.613598 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260619 02:52:23.738338 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:23.738002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:37625)
0619 02:52:23.738244 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:23.738284 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260619 02:52:24.551236 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:24.550980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.193:38379)
0619 02:52:24.551147 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:24.551187 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260619 02:52:24.614508 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:24.614254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.194:53697)
0619 02:52:24.614416 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:24.614460 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260619 02:52:24.732605 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 4528
I20260619 02:52:24.736752 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 4690
I20260619 02:52:24.739105 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:24.738871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45997 (local address 127.21.164.195:33947)
0619 02:52:24.739031 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:24.739064 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260619 02:52:24.740760 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 4824
2026-06-19T02:52:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17967 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:52:24 dist-test-slave-jt0t krb5kdc[5011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:52:24 dist-test-slave-jt0t krb5kdc[5011](info): set up 2 sockets
Jun 19 02:52:24 dist-test-slave-jt0t krb5kdc[5011](info): commencing operation
krb5kdc: starting...
W20260619 02:52:25.692472 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:52:25 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:52:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:52:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:52:25.802685 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34175
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:34175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:25.883302 5024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:25.883579 5024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:25.883613 5024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:25.884828 5024 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:52:25.884864 5024 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:52:25.884876 5024 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:52:25.884886 5024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:25.884900 5024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:25.884923 5024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:25.884945 5024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:25.884965 5024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:25.886500 5024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:34175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:25.886729 5024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:25.886920 5024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:25.891811 5032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:25.891834 5029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:25.891887 5030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:25.891961 5024 server_base.cc:1034] running on GCE node
I20260619 02:52:25.892326 5024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:25.892611 5024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:25.893810 5024 hybrid_clock.cc:648] HybridClock initialized: now 1781837545893790 us; error 33 us; skew 500 ppm
Jun 19 02:52:25 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:25.896221 5024 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:52:25.896713 5024 webserver.cc:460] Webserver started at http://127.21.164.254:42673/ using document root <none> and password file <none>
I20260619 02:52:25.896944 5024 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:25.896994 5024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:25.897118 5024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:25.898216 5024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "78e79060646d4f7dbcf1134384fb9579"
format_stamp: "Formatted at 2026-06-19 02:52:25 on dist-test-slave-jt0t"
server_key: "3c5722996045788975fba565f9c585e7"
server_key_iv: "222b07f23250e0748a5b72c6aa675c46"
server_key_version: "encryptionkey@0"
I20260619 02:52:25.898558 5024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "78e79060646d4f7dbcf1134384fb9579"
format_stamp: "Formatted at 2026-06-19 02:52:25 on dist-test-slave-jt0t"
server_key: "3c5722996045788975fba565f9c585e7"
server_key_iv: "222b07f23250e0748a5b72c6aa675c46"
server_key_version: "encryptionkey@0"
I20260619 02:52:25.899806 5024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:52:25.900513 5039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:25.900745 5024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:25.900826 5024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "78e79060646d4f7dbcf1134384fb9579"
format_stamp: "Formatted at 2026-06-19 02:52:25 on dist-test-slave-jt0t"
server_key: "3c5722996045788975fba565f9c585e7"
server_key_iv: "222b07f23250e0748a5b72c6aa675c46"
server_key_version: "encryptionkey@0"
I20260619 02:52:25.900897 5024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:25.934513 5024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:25.936383 5024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:25.936535 5024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:25.940315 5024 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:34175
I20260619 02:52:25.940359 5091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:34175 every 8 connection(s)
I20260619 02:52:25.940704 5024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:25.941525 5092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:25.943771 5092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Bootstrap starting.
I20260619 02:52:25.944309 5092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:25.944546 5092 log.cc:824] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:25.945192 5092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: No bootstrap required, opened a new log
I20260619 02:52:25.946791 5092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:52:25.946926 5092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:25.946947 5092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e79060646d4f7dbcf1134384fb9579, State: Initialized, Role: FOLLOWER
I20260619 02:52:25.947055 5092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:52:25.947108 5092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:25.947135 5092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:25.947175 5092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:25.947425 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 5024
I20260619 02:52:25.947520 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:52:25.947681 22163 external_mini_cluster.cc:1428] Setting key 167d08b34a6f52a35fd18f4fd3efafcd
I20260619 02:52:25.947708 5092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:52:25.947813 5092 leader_election.cc:302] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78e79060646d4f7dbcf1134384fb9579; no voters:
I20260619 02:52:25.947960 5092 leader_election.cc:288] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:52:25.948056 5095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:25.948226 5092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:25.948463 5095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 1 LEADER]: Becoming Leader. State: Replica: 78e79060646d4f7dbcf1134384fb9579, State: Running, Role: LEADER
I20260619 02:52:25.948602 5095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:52:25.948966 5095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78e79060646d4f7dbcf1134384fb9579. Latest consensus state: current_term: 1 leader_uuid: "78e79060646d4f7dbcf1134384fb9579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } } }
I20260619 02:52:25.949046 5095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:25.949034 5096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78e79060646d4f7dbcf1134384fb9579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } } }
I20260619 02:52:25.949122 5096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: This master's current role is: LEADER
W20260619 02:52:25.949991 5112 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:52:25.950042 5112 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:52:25.950357 5106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:25.950570 5106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:25.952188 5106 catalog_manager.cc:1347] Generated new cluster ID: 655b200cc9894ae9a13cf4bfd39d4651
I20260619 02:52:25.952239 5106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:52:25 dist-test-slave-jt0t krb5kdc[5011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:25.958885 5106 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:52:25.959260 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:25.948126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.21.164.254:34175)
0619 02:52:25.948281 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:25.948292 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:25.948603 (+ 311us) server_negotiation.cc:407] Connection header received
0619 02:52:25.948746 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:25.948755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:25.948817 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:25.948952 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:25.949805 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:25.951191 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:25.952120 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:25.952742 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:25.952844 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:25.955548 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:25.955553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:25.955553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:25.955565 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:25.957483 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:25.957864 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:25.957867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:25.957867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:25.957994 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:25.958189 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:25.958189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:25.958190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:25.958479 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:25.958617 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:25.958876 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 02:52:25.959035 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260619 02:52:25.959745 5106 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:25.967959 5106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Generated new TSK 0
I20260619 02:52:25.968147 5106 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:52:25.993396 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34175
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:26.074246 5118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:26.074465 5118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:26.074499 5118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:26.075657 5118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:26.075699 5118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:26.075736 5118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:52:26.077207 5118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:26.077443 5118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:26.077659 5118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:26.082638 5126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:26.082617 5124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:26.082651 5118 server_base.cc:1034] running on GCE node
W20260619 02:52:26.082617 5123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:26.083035 5118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:26.083256 5118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:26.084414 5118 hybrid_clock.cc:648] HybridClock initialized: now 1781837546084400 us; error 30 us; skew 500 ppm
Jun 19 02:52:26 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:26.086956 5118 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:52:26.087390 5118 webserver.cc:460] Webserver started at http://127.21.164.193:38043/ using document root <none> and password file <none>
I20260619 02:52:26.087569 5118 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:26.087608 5118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:26.087700 5118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:26.088768 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "ef99a8f4f6c6462387cc2e4b63d2e514"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "8aa086c95d08d5bb86179348b00ce95b"
server_key_iv: "440bd6d86a0bace099a2e133e7b00853"
server_key_version: "encryptionkey@0"
I20260619 02:52:26.089080 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "ef99a8f4f6c6462387cc2e4b63d2e514"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "8aa086c95d08d5bb86179348b00ce95b"
server_key_iv: "440bd6d86a0bace099a2e133e7b00853"
server_key_version: "encryptionkey@0"
I20260619 02:52:26.090250 5118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:26.090865 5133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:26.091012 5118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:52:26.091104 5118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "ef99a8f4f6c6462387cc2e4b63d2e514"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "8aa086c95d08d5bb86179348b00ce95b"
server_key_iv: "440bd6d86a0bace099a2e133e7b00853"
server_key_version: "encryptionkey@0"
I20260619 02:52:26.091174 5118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:26.103977 5118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:26.106163 5118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:26.106293 5118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:26.106523 5118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:26.106830 5118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:26.106864 5118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:26.106904 5118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:26.106933 5118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:26.112393 5118 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:45255
I20260619 02:52:26.112455 5246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:45255 every 8 connection(s)
I20260619 02:52:26.112766 5118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:26.118443 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 5118
Jun 19 02:52:26 dist-test-slave-jt0t krb5kdc[5011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:26.118551 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:52:26.118733 22163 external_mini_cluster.cc:1428] Setting key a08aace37722ff91ac3db9629a26c371
I20260619 02:52:26.122014 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.113502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:38377 (local address 127.21.164.254:34175)
0619 02:52:26.113563 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:26.113565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:26.113749 (+ 184us) server_negotiation.cc:407] Connection header received
0619 02:52:26.114212 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:26.114213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:26.114226 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:26.114282 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:26.115234 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.115731 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.116665 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.116877 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.116926 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.118978 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:26.118980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:26.118980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:26.118998 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:26.120524 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:26.121066 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:26.121067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:26.121067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:26.121121 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:26.121365 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:26.121366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:26.121366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:26.121468 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:26.121514 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:26.121844 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 02:52:26.121929 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:52:26.122293 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.113474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:38377)
0619 02:52:26.113631 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.113672 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:26.114139 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:26.114340 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:26.114344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:26.114946 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:26.115146 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.115151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.115794 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.115795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.116583 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.116585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.116951 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.116952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.117053 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.117535 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:26.117543 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:26.118855 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:26.120619 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:26.120625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:26.120627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:26.120984 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:26.121178 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:26.121179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:26.121179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:26.121310 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:26.121519 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:26.121524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:26.121692 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 02:52:26.121808 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 02:52:26.122063 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260619 02:52:26.122805 5247 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34175
I20260619 02:52:26.122920 5247 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:26.123116 5247 heartbeater.cc:511] Master 127.21.164.254:34175 requested a full tablet report, sending...
I20260619 02:52:26.123699 5056 ts_manager.cc:194] Registered new tserver with Master: ef99a8f4f6c6462387cc2e4b63d2e514 (127.21.164.193:45255)
I20260619 02:52:26.124294 5056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:38377
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:52:26.143589 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34175
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:26.223467 5252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:26.223688 5252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:26.223711 5252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:26.224902 5252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:26.224942 5252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:26.224968 5252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:52:26.226387 5252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:26.226611 5252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:26.226820 5252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:26.231595 5257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:26.231619 5258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:26.231654 5252 server_base.cc:1034] running on GCE node
W20260619 02:52:26.231619 5260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:26.231907 5252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:26.232124 5252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:26.233266 5252 hybrid_clock.cc:648] HybridClock initialized: now 1781837546233245 us; error 30 us; skew 500 ppm
Jun 19 02:52:26 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:26.235661 5252 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:52:26.236140 5252 webserver.cc:460] Webserver started at http://127.21.164.194:32997/ using document root <none> and password file <none>
I20260619 02:52:26.236346 5252 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:26.236394 5252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:26.236521 5252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:26.237601 5252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "a33a3a4f7e4c4ef2b3354648215f8974"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "4aadfd5f575199e2b8e99566dd623ea5"
server_key_iv: "a0b5c7276fe12b483819daf8b3f1361a"
server_key_version: "encryptionkey@0"
I20260619 02:52:26.237942 5252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "a33a3a4f7e4c4ef2b3354648215f8974"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "4aadfd5f575199e2b8e99566dd623ea5"
server_key_iv: "a0b5c7276fe12b483819daf8b3f1361a"
server_key_version: "encryptionkey@0"
I20260619 02:52:26.239161 5252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:52:26.239828 5267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:26.239980 5252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:52:26.240060 5252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "a33a3a4f7e4c4ef2b3354648215f8974"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "4aadfd5f575199e2b8e99566dd623ea5"
server_key_iv: "a0b5c7276fe12b483819daf8b3f1361a"
server_key_version: "encryptionkey@0"
I20260619 02:52:26.240125 5252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:26.248025 5252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:26.250195 5252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:26.250325 5252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:26.250542 5252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:26.250850 5252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:26.250883 5252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:26.250912 5252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:26.250933 5252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:26.256331 5252 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:36031
I20260619 02:52:26.256372 5380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:36031 every 8 connection(s)
I20260619 02:52:26.256721 5252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:52:26.258247 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 5252
I20260619 02:52:26.258342 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:52:26.258517 22163 external_mini_cluster.cc:1428] Setting key 6087d7757d7bb3c892c3bf4cf748148f
Jun 19 02:52:26 dist-test-slave-jt0t krb5kdc[5011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:26.266741 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.257423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:42785 (local address 127.21.164.254:34175)
0619 02:52:26.257481 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:26.257483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:26.257673 (+ 190us) server_negotiation.cc:407] Connection header received
0619 02:52:26.258151 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:26.258153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:26.258166 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:26.258248 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:26.259283 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.259955 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.261322 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.261535 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.261601 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.263919 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:26.263921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:26.263921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:26.263931 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:26.265454 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:26.265834 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:26.265836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:26.265836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:26.265893 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:26.266104 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:26.266104 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:26.266105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:26.266246 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:26.266322 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:26.266551 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 02:52:26.266641 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
I20260619 02:52:26.266872 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.257396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:42785)
0619 02:52:26.257542 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.257592 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:26.258071 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:26.258306 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:26.258310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:26.258952 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:26.259206 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.259212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.260062 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.260063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.261227 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.261230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.261615 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.261616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.261739 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.262261 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:26.262272 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:26.263826 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:26.265535 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:26.265541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:26.265543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:26.265762 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:26.265955 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:26.265956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:26.265956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:26.266051 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:26.266305 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:26.266308 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:26.266430 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 02:52:26.266508 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:52:26.266631 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260619 02:52:26.267302 5381 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34175
I20260619 02:52:26.267395 5381 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:26.267581 5381 heartbeater.cc:511] Master 127.21.164.254:34175 requested a full tablet report, sending...
I20260619 02:52:26.268042 5056 ts_manager.cc:194] Registered new tserver with Master: a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031)
I20260619 02:52:26.268374 5056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:42785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:52:26.283416 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34175
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:26.362469 5386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:26.362668 5386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:26.362690 5386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:26.363804 5386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:26.363847 5386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:26.363873 5386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:52:26.365301 5386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:26.365525 5386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:26.365741 5386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:52:26.370478 5386 server_base.cc:1034] running on GCE node
W20260619 02:52:26.370467 5394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:26.370456 5392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:26.370462 5391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:26.370858 5386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:26.371078 5386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:26.372227 5386 hybrid_clock.cc:648] HybridClock initialized: now 1781837546372209 us; error 32 us; skew 500 ppm
Jun 19 02:52:26 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:26.374691 5386 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:52:26.375123 5386 webserver.cc:460] Webserver started at http://127.21.164.195:34245/ using document root <none> and password file <none>
I20260619 02:52:26.375312 5386 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:26.375355 5386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:26.375474 5386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:52:26.376521 5386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "e0bba9c3872d4add9b94166ad19fb1d4"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "4c81767070fc632fbd2b30306ed38560"
server_key_iv: "d508a804c714bc74831fa9852ea954a8"
server_key_version: "encryptionkey@0"
I20260619 02:52:26.376879 5386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "e0bba9c3872d4add9b94166ad19fb1d4"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "4c81767070fc632fbd2b30306ed38560"
server_key_iv: "d508a804c714bc74831fa9852ea954a8"
server_key_version: "encryptionkey@0"
I20260619 02:52:26.378077 5386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:26.378711 5401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:26.378871 5386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:52:26.378943 5386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "e0bba9c3872d4add9b94166ad19fb1d4"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "4c81767070fc632fbd2b30306ed38560"
server_key_iv: "d508a804c714bc74831fa9852ea954a8"
server_key_version: "encryptionkey@0"
I20260619 02:52:26.379022 5386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:26.389586 5386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:26.391263 5386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:26.391386 5386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:26.391585 5386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:26.391880 5386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:26.391913 5386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:26.391934 5386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:26.391947 5386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:26.397693 5386 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:45423
I20260619 02:52:26.397738 5514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:45423 every 8 connection(s)
I20260619 02:52:26.398069 5386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:52:26 dist-test-slave-jt0t krb5kdc[5011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837546, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:26.406518 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.398712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:47297 (local address 127.21.164.254:34175)
0619 02:52:26.398778 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:26.398780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:26.398966 (+ 186us) server_negotiation.cc:407] Connection header received
0619 02:52:26.399452 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:26.399454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:26.399464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:26.399507 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:26.400390 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.400947 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.401911 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.402096 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.402140 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.404235 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:26.404236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:26.404237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:26.404243 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:26.405357 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:26.405713 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:26.405714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:26.405714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:26.405758 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:26.405943 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:26.405943 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:26.405943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:26.406062 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:26.406109 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:26.406391 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 02:52:26.406447 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:52:26.406746 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.398685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:47297)
0619 02:52:26.398845 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.398887 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:26.399357 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:26.399570 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:26.399573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:26.400120 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:26.400311 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.400316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.401035 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.401036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.401848 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.401851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.402149 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.402149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.402257 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.402763 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:26.402772 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:26.404150 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:26.405420 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:26.405426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:26.405429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:26.405656 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:26.405804 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:26.405805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:26.405805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:26.405901 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:26.406140 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:26.406146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:26.406270 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 02:52:26.406374 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 02:52:26.406504 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260619 02:52:26.407303 5515 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34175
I20260619 02:52:26.407410 5515 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:26.407677 5515 heartbeater.cc:511] Master 127.21.164.254:34175 requested a full tablet report, sending...
I20260619 02:52:26.408130 5056 ts_manager.cc:194] Registered new tserver with Master: e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423)
I20260619 02:52:26.408144 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 5386
I20260619 02:52:26.408241 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:52:26.408424 22163 external_mini_cluster.cc:1428] Setting key 66ab5c5a5ad6490597011a1a44f9af4a
I20260619 02:52:26.408507 5056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:47297
I20260619 02:52:26.409729 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:52:26.416416 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.410945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.21.164.254:34175)
0619 02:52:26.411005 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:26.411007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:26.411142 (+ 135us) server_negotiation.cc:407] Connection header received
0619 02:52:26.411204 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:26.411204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:26.411215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:26.411255 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:26.411774 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.412251 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.413010 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.413213 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.413266 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.414370 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:26.414372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:26.414372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:26.414379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:26.415396 (+ 1017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:26.415701 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:26.415703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:26.415703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:26.415756 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:26.415904 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:26.415905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:26.415905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:26.416051 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:26.416140 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:26.416250 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 02:52:26.416326 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:52:26.417810 5056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:52:26.419323 5056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:52:26.425366 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.421342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:45255 (local address 127.0.0.1:57768)
0619 02:52:26.421718 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.421725 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:26.421752 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:26.422073 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:26.422074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:26.422076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:26.422114 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.422115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.423046 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.423047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.424226 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.424229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.425096 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.425097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.425180 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.425192 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 02:52:26.425251 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:52:26.425291 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":86,"threads_started":1}
I20260619 02:52:26.425401 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.421764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.21.164.193:45255)
0619 02:52:26.421916 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:26.421921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:26.421930 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:52:26.421962 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:26.421965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:26.421979 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:26.422178 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:26.422244 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.422947 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.424306 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.425015 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.425085 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.425138 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:26.425278 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 02:52:26.425319 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260619 02:52:26.425565 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.421339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:45423 (local address 127.0.0.1:53300)
0619 02:52:26.421521 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.421562 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:26.421624 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:26.421913 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:26.421916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:26.421918 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:26.422020 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.422022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.423430 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.423431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.424507 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.424509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.425361 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.425362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.425416 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.425420 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:52:26.425469 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 02:52:26.425504 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":24,"thread_start_us":75,"threads_started":1}
I20260619 02:52:26.425760 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.421336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:36031 (local address 127.0.0.1:41778)
0619 02:52:26.421521 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.421562 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:26.421624 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:26.422109 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:26.422110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:26.422111 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:26.422137 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.422138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.423422 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.423423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.424320 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.424322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.425561 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.425562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.425619 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.425622 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:52:26.425670 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 02:52:26.425703 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":36,"thread_start_us":75,"threads_started":1}
I20260619 02:52:26.425827 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.421565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.21.164.195:45423)
0619 02:52:26.421734 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:26.421740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:26.421750 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:52:26.421790 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:26.421793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:26.421811 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:26.421896 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:26.422261 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.423340 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.424581 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.425283 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.425350 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.425396 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:26.425707 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 02:52:26.425752 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260619 02:52:26.426029 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.421588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.21.164.194:36031)
0619 02:52:26.421977 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:26.421980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:26.421990 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:52:26.422026 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:26.422028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:26.422043 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:26.422116 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:26.422179 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.423305 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.424397 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.425016 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.425082 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.425138 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:26.425908 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 02:52:26.425956 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":80,"threads_started":1}
I20260619 02:52:26.426404 5181 tablet_service.cc:1467] Processing CreateTablet for tablet d37f5068478a473eb1f4cfde79f15e30 (DEFAULT_TABLE table=test-table [id=65806bb3bf874271b6acdc2877a1c292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:26.426465 5449 tablet_service.cc:1467] Processing CreateTablet for tablet d37f5068478a473eb1f4cfde79f15e30 (DEFAULT_TABLE table=test-table [id=65806bb3bf874271b6acdc2877a1c292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:26.426772 5181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37f5068478a473eb1f4cfde79f15e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:26.426779 5449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37f5068478a473eb1f4cfde79f15e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:26.429291 5529 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Bootstrap starting.
I20260619 02:52:26.429814 5530 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: Bootstrap starting.
I20260619 02:52:26.430032 5529 tablet_bootstrap.cc:654] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:26.429963 5315 tablet_service.cc:1467] Processing CreateTablet for tablet d37f5068478a473eb1f4cfde79f15e30 (DEFAULT_TABLE table=test-table [id=65806bb3bf874271b6acdc2877a1c292]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:26.430258 5315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37f5068478a473eb1f4cfde79f15e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:26.430418 5530 tablet_bootstrap.cc:654] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:26.430646 5529 log.cc:824] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:26.430968 5530 log.cc:824] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:26.431404 5529 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: No bootstrap required, opened a new log
I20260619 02:52:26.431463 5529 ts_tablet_manager.cc:1397] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:26.432857 5529 raft_consensus.cc:348] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:26.432987 5533 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Bootstrap starting.
I20260619 02:52:26.433040 5529 raft_consensus.cc:374] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:26.433068 5529 raft_consensus.cc:729] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef99a8f4f6c6462387cc2e4b63d2e514, State: Initialized, Role: FOLLOWER
I20260619 02:52:26.433179 5529 consensus_queue.cc:260] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:26.433439 5529 ts_tablet_manager.cc:1428] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:26.433460 5247 heartbeater.cc:503] Master 127.21.164.254:34175 was elected leader, sending a full tablet report...
I20260619 02:52:26.433573 5533 tablet_bootstrap.cc:654] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:26.434026 5533 log.cc:824] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:26.434178 5530 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: No bootstrap required, opened a new log
I20260619 02:52:26.434249 5530 ts_tablet_manager.cc:1397] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:52:26.434815 5533 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: No bootstrap required, opened a new log
I20260619 02:52:26.434871 5533 ts_tablet_manager.cc:1397] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:52:26.435599 5530 raft_consensus.cc:348] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:26.435726 5530 raft_consensus.cc:374] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:26.435746 5530 raft_consensus.cc:729] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0bba9c3872d4add9b94166ad19fb1d4, State: Initialized, Role: FOLLOWER
I20260619 02:52:26.435827 5530 consensus_queue.cc:260] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:26.436074 5530 ts_tablet_manager.cc:1428] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:26.436116 5515 heartbeater.cc:503] Master 127.21.164.254:34175 was elected leader, sending a full tablet report...
I20260619 02:52:26.436398 5533 raft_consensus.cc:348] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:26.436563 5533 raft_consensus.cc:374] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:26.436611 5533 raft_consensus.cc:729] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33a3a4f7e4c4ef2b3354648215f8974, State: Initialized, Role: FOLLOWER
I20260619 02:52:26.436756 5533 consensus_queue.cc:260] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:26.437044 5381 heartbeater.cc:503] Master 127.21.164.254:34175 was elected leader, sending a full tablet report...
I20260619 02:52:26.437041 5533 ts_tablet_manager.cc:1428] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260619 02:52:26.507270 5382 tablet.cc:2369] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:52:26.613477 5248 tablet.cc:2369] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:52:26.648584 5516 tablet.cc:2369] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:26.833806 5534 raft_consensus.cc:482] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:26.833973 5534 raft_consensus.cc:504] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:26.834436 5534 leader_election.cc:288] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031), e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423)
I20260619 02:52:26.838893 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.834745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:58493 (local address 127.21.164.194:36031)
0619 02:52:26.834806 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:26.834808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:26.834913 (+ 105us) server_negotiation.cc:407] Connection header received
0619 02:52:26.834986 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:26.834987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:26.835000 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:26.835066 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:26.835352 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.836162 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.837845 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.838445 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.838527 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.838538 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:26.838758 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 02:52:26.838808 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 02:52:26.838893 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.834800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:39095 (local address 127.21.164.195:45423)
0619 02:52:26.834849 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:26.834850 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:26.834915 (+ 65us) server_negotiation.cc:407] Connection header received
0619 02:52:26.835005 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:26.835007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:26.835015 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:26.835066 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:26.835354 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.836166 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.837845 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.838453 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.838527 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.838538 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:26.838758 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 02:52:26.838808 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 02:52:26.838924 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.834669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:36031 (local address 127.21.164.193:58493)
0619 02:52:26.834824 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.834837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:26.834920 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:26.835124 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:26.835126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:26.835130 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:26.835269 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.835272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.836265 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.836266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.837734 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.837737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.838558 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.838559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.838665 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.838673 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:52:26.838740 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 02:52:26.838798 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":60,"thread_start_us":79,"threads_started":1}
I20260619 02:52:26.838924 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.834669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:45423 (local address 127.21.164.193:39095)
0619 02:52:26.834830 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.834841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:26.834920 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:26.835124 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:26.835126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:26.835130 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:26.835276 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.835277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.836266 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.836267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.837734 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:26.837737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.838556 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.838558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:26.838665 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.838674 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:52:26.838740 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:52:26.838798 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":118,"thread_start_us":67,"threads_started":1}
I20260619 02:52:26.839370 5469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37f5068478a473eb1f4cfde79f15e30" candidate_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" is_pre_election: true
I20260619 02:52:26.839550 5469 raft_consensus.cc:2454] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef99a8f4f6c6462387cc2e4b63d2e514 in term 0.
I20260619 02:52:26.839504 5335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37f5068478a473eb1f4cfde79f15e30" candidate_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" is_pre_election: true
I20260619 02:52:26.839672 5335 raft_consensus.cc:2454] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef99a8f4f6c6462387cc2e4b63d2e514 in term 0.
I20260619 02:52:26.839778 5135 leader_election.cc:302] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0bba9c3872d4add9b94166ad19fb1d4, ef99a8f4f6c6462387cc2e4b63d2e514; no voters:
I20260619 02:52:26.839919 5534 raft_consensus.cc:2781] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:26.839984 5534 raft_consensus.cc:482] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:26.840003 5534 raft_consensus.cc:3037] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:26.840934 5534 raft_consensus.cc:504] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:26.841061 5534 leader_election.cc:288] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 1 election: Requested vote from peers a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031), e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423)
I20260619 02:52:26.841240 5335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37f5068478a473eb1f4cfde79f15e30" candidate_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33a3a4f7e4c4ef2b3354648215f8974"
I20260619 02:52:26.841269 5469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37f5068478a473eb1f4cfde79f15e30" candidate_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bba9c3872d4add9b94166ad19fb1d4"
I20260619 02:52:26.841321 5335 raft_consensus.cc:3037] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:26.841338 5469 raft_consensus.cc:3037] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:26.842123 5335 raft_consensus.cc:2454] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef99a8f4f6c6462387cc2e4b63d2e514 in term 1.
I20260619 02:52:26.842137 5469 raft_consensus.cc:2454] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef99a8f4f6c6462387cc2e4b63d2e514 in term 1.
I20260619 02:52:26.842310 5137 leader_election.cc:302] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a33a3a4f7e4c4ef2b3354648215f8974, ef99a8f4f6c6462387cc2e4b63d2e514; no voters:
I20260619 02:52:26.842391 5534 raft_consensus.cc:2781] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:26.842555 5534 raft_consensus.cc:686] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 LEADER]: Becoming Leader. State: Replica: ef99a8f4f6c6462387cc2e4b63d2e514, State: Running, Role: LEADER
I20260619 02:52:26.842690 5534 consensus_queue.cc:237] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:26.843672 5055 catalog_manager.cc:5495] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef99a8f4f6c6462387cc2e4b63d2e514 (127.21.164.193). New cstate: current_term: 1 leader_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } health_report { overall_health: UNKNOWN } } }
I20260619 02:52:26.864681 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.862353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.21.164.193:45255)
0619 02:52:26.862408 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:26.862410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:26.862419 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:52:26.862460 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:26.862460 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:26.862478 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:26.862543 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:26.862697 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.863207 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.864028 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:26.864226 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:26.864282 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:26.864389 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:26.864497 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:26.864561 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:26.864572 (+ 11us) server_negotiation.cc:299] Negotiation successful
0619 02:52:26.864589 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:52:26.866542 5469 raft_consensus.cc:1261] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4 [term 1 FOLLOWER]: Refusing update from remote peer ef99a8f4f6c6462387cc2e4b63d2e514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:26.866540 5335 raft_consensus.cc:1261] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 FOLLOWER]: Refusing update from remote peer ef99a8f4f6c6462387cc2e4b63d2e514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:26.866883 5540 consensus_queue.cc:1035] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:26.867026 5534 consensus_queue.cc:1035] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:26.869709 5545 mvcc.cc:205] Tried to move back new op lower bound from 7298406591962755072 to 7298406591868289024. Current Snapshot: MvccSnapshot[applied={T|T < 7298406591962755072}]
I20260619 02:52:26.870035 5544 mvcc.cc:205] Tried to move back new op lower bound from 7298406591962755072 to 7298406591868289024. Current Snapshot: MvccSnapshot[applied={T|T < 7298406591962755072}]
I20260619 02:52:26.871738 5055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:26.871825 5055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:26.872591 5055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Sending DeleteTablet for 3 replicas of tablet d37f5068478a473eb1f4cfde79f15e30
I20260619 02:52:26.872946 5181 tablet_service.cc:1514] Processing DeleteTablet for tablet d37f5068478a473eb1f4cfde79f15e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:26 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:57768
I20260619 02:52:26.872938 5315 tablet_service.cc:1514] Processing DeleteTablet for tablet d37f5068478a473eb1f4cfde79f15e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:26 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:41778
I20260619 02:52:26.873170 5449 tablet_service.cc:1514] Processing DeleteTablet for tablet d37f5068478a473eb1f4cfde79f15e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:26 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:53300
I20260619 02:52:26.873237 5553 tablet_replica.cc:330] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: stopping tablet replica
I20260619 02:52:26.873291 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 5118
I20260619 02:52:26.873337 5553 raft_consensus.cc:2229] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:26.874269 5555 tablet_replica.cc:330] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: stopping tablet replica
I20260619 02:52:26.874372 5555 raft_consensus.cc:2229] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:26.874475 5555 raft_consensus.cc:2258] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:26.874905 5555 ts_tablet_manager.cc:1905] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:26.876492 5555 ts_tablet_manager.cc:1918] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:26.876551 5555 log.cc:1196] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/d37f5068478a473eb1f4cfde79f15e30
I20260619 02:52:26.876833 5555 ts_tablet_manager.cc:1939] T d37f5068478a473eb1f4cfde79f15e30 P e0bba9c3872d4add9b94166ad19fb1d4: Deleting consensus metadata
I20260619 02:52:26.877231 5041 catalog_manager.cc:4841] TS e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423): tablet d37f5068478a473eb1f4cfde79f15e30 (table test-table [id=65806bb3bf874271b6acdc2877a1c292]) successfully deleted
W20260619 02:52:26.878325 5042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:45255 (error 108)
I20260619 02:52:26.878835 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.878684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:45255 (local address 127.0.0.1:57786)
0619 02:52:26.878745 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.878795 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 02:52:26.878932 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 5252
I20260619 02:52:26.878965 5553 raft_consensus.cc:2258] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 02:52:26.880326 5042 catalog_manager.cc:4584] TS ef99a8f4f6c6462387cc2e4b63d2e514 (127.21.164.193:45255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d37f5068478a473eb1f4cfde79f15e30: Network error: Client connection negotiation failed: client connection to 127.21.164.193:45255: connect: Connection refused (error 111)
I20260619 02:52:26.884796 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 5386
I20260619 02:52:26.884912 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:26.884775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:36031 (local address 127.0.0.1:41790)
0619 02:52:26.884816 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:26.884860 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.194:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 02:52:26.890157 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 5024
I20260619 02:52:46.895493 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34175
--webserver_interface=127.21.164.254
--webserver_port=42673
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:34175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:52:46.983330 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:46.983556 5557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:46.983577 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:46.984783 5557 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:52:46.984823 5557 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:52:46.984834 5557 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:52:46.984844 5557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:52:46.984856 5557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:46.984867 5557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:46.984879 5557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:52:46.984894 5557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:52:46.986460 5557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:34175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=42673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:46.986708 5557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:46.986948 5557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:46.995908 5566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:46.995924 5563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:46.995949 5557 server_base.cc:1034] running on GCE node
W20260619 02:52:46.996076 5564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:46.996543 5557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:46.996917 5557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:46.998080 5557 hybrid_clock.cc:648] HybridClock initialized: now 1781837566998068 us; error 30 us; skew 500 ppm
Jun 19 02:52:47 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:47.005494 5557 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:52:47.005955 5557 webserver.cc:460] Webserver started at http://127.21.164.254:42673/ using document root <none> and password file <none>
I20260619 02:52:47.006204 5557 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:47.006263 5557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:47.007740 5557 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:47.008395 5573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:47.008553 5557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:47.008641 5557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "78e79060646d4f7dbcf1134384fb9579"
format_stamp: "Formatted at 2026-06-19 02:52:25 on dist-test-slave-jt0t"
server_key: "3c5722996045788975fba565f9c585e7"
server_key_iv: "222b07f23250e0748a5b72c6aa675c46"
server_key_version: "encryptionkey@0"
I20260619 02:52:47.009130 5557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:47.018649 5557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:47.020303 5557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:47.020429 5557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:47.024716 5557 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:34175
I20260619 02:52:47.024727 5625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:34175 every 8 connection(s)
I20260619 02:52:47.025173 5557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:52:47.026741 5626 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:52:47.027459 5626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Bootstrap starting.
I20260619 02:52:47.031235 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 5557
I20260619 02:52:47.031625 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:45255
--local_ip_for_outbound_sockets=127.21.164.193
--tserver_master_addrs=127.21.164.254:34175
--webserver_port=38043
--webserver_interface=127.21.164.193
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:47.037771 5626 log.cc:824] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:47.039855 5626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:47.040177 5626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Bootstrap complete.
I20260619 02:52:47.042546 5626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:52:47.042797 5626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e79060646d4f7dbcf1134384fb9579, State: Initialized, Role: FOLLOWER
I20260619 02:52:47.042960 5626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:52:47.043028 5626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:52:47.043076 5626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:52:47.043126 5626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:52:47.044303 5626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:52:47.044399 5626 leader_election.cc:302] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78e79060646d4f7dbcf1134384fb9579; no voters:
I20260619 02:52:47.044580 5626 leader_election.cc:288] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:52:47.044629 5630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:52:47.044790 5630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 2 LEADER]: Becoming Leader. State: Replica: 78e79060646d4f7dbcf1134384fb9579, State: Running, Role: LEADER
I20260619 02:52:47.044895 5626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:52:47.044950 5630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:52:47.045301 5630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78e79060646d4f7dbcf1134384fb9579. Latest consensus state: current_term: 2 leader_uuid: "78e79060646d4f7dbcf1134384fb9579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } } }
I20260619 02:52:47.045454 5630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:47.045596 5632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "78e79060646d4f7dbcf1134384fb9579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } } }
I20260619 02:52:47.045686 5632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: This master's current role is: LEADER
I20260619 02:52:47.045862 5638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:52:47.046854 5638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:52:47.047019 5638 catalog_manager.cc:1259] Loaded cluster ID: 655b200cc9894ae9a13cf4bfd39d4651
I20260619 02:52:47.047242 5638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:52:47.048425 5638 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:52:47.048602 5638 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Loaded TSK: 0
I20260619 02:52:47.061890 5638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Generated new TSK 1
I20260619 02:52:47.062011 5638 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:52:47.124063 5628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:47.124295 5628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:47.124326 5628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:47.125473 5628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:47.125512 5628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:47.125547 5628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:52:47.126976 5628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:45255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=38043
--tserver_master_addrs=127.21.164.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:47.127220 5628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:47.127460 5628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:47.133971 5654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:47.134087 5653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:47.134229 5628 server_base.cc:1034] running on GCE node
W20260619 02:52:47.133970 5656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:47.134479 5628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:47.134788 5628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:47.135977 5628 hybrid_clock.cc:648] HybridClock initialized: now 1781837567135956 us; error 38 us; skew 500 ppm
Jun 19 02:52:47 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:47.139117 5628 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:52:47.139669 5628 webserver.cc:460] Webserver started at http://127.21.164.193:38043/ using document root <none> and password file <none>
I20260619 02:52:47.139933 5628 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:47.140008 5628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:47.141835 5628 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:47.142591 5663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:47.142797 5628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:47.142884 5628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "ef99a8f4f6c6462387cc2e4b63d2e514"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "8aa086c95d08d5bb86179348b00ce95b"
server_key_iv: "440bd6d86a0bace099a2e133e7b00853"
server_key_version: "encryptionkey@0"
I20260619 02:52:47.143254 5628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:47.159004 5628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:47.161274 5628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:47.161403 5628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:47.161672 5628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:47.162161 5670 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:52:47.164263 5628 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:52:47.164317 5628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 02:52:47.164354 5628 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:52:47.165150 5628 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:52:47.165182 5628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 02:52:47.165264 5670 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Bootstrap starting.
I20260619 02:52:47.172888 5628 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:45255
I20260619 02:52:47.172961 5777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:45255 every 8 connection(s)
I20260619 02:52:47.173386 5628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:52:47.178109 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 5628
I20260619 02:52:47.178468 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:36031
--local_ip_for_outbound_sockets=127.21.164.194
--tserver_master_addrs=127.21.164.254:34175
--webserver_port=32997
--webserver_interface=127.21.164.194
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:47.178607 5670 log.cc:824] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:47.181617 5670 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:47.181951 5670 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Bootstrap complete.
I20260619 02:52:47.183106 5670 ts_tablet_manager.cc:1397] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.013s
Jun 19 02:52:47 dist-test-slave-jt0t krb5kdc[5011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:47.184546 5670 raft_consensus.cc:348] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:47.184969 5670 raft_consensus.cc:729] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef99a8f4f6c6462387cc2e4b63d2e514, State: Initialized, Role: FOLLOWER
I20260619 02:52:47.185150 5670 consensus_queue.cc:260] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:47.185509 5670 ts_tablet_manager.cc:1428] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260619 02:52:47.188997 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.174292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:35481 (local address 127.21.164.254:34175)
0619 02:52:47.176859 (+ 2567us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.176871 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.176931 (+ 60us) server_negotiation.cc:407] Connection header received
0619 02:52:47.177086 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.177093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.177149 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.177307 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:47.178107 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.179474 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.181040 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.181460 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.181584 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.184532 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:47.184538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:47.184538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:47.184553 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:47.186784 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.187364 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.187369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.187369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.187506 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.187803 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.187804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.187805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.188143 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:47.188251 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:47.188679 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 02:52:47.188814 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2500,"thread_start_us":113,"threads_started":1}
I20260619 02:52:47.189043 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.174196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:35481)
0619 02:52:47.174395 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:47.174445 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:47.175005 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:47.177355 (+ 2350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:47.177360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:47.177687 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:47.177938 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.177948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.179622 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.179623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.180942 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.180946 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.181582 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.181583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.181739 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.182384 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:47.182404 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:47.184385 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:47.186899 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:47.186907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:47.186909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:47.187256 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:47.187586 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:47.187587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:47.187588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:47.187721 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:47.188286 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:47.188294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:47.188465 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 02:52:47.188597 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 02:52:47.188834 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260619 02:52:47.189898 5778 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34175
I20260619 02:52:47.189997 5778 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:47.190178 5778 heartbeater.cc:511] Master 127.21.164.254:34175 requested a full tablet report, sending...
I20260619 02:52:47.190970 5590 ts_manager.cc:194] Registered new tserver with Master: ef99a8f4f6c6462387cc2e4b63d2e514 (127.21.164.193:45255)
I20260619 02:52:47.192116 5590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:35481
I20260619 02:52:47.193172 5778 heartbeater.cc:503] Master 127.21.164.254:34175 was elected leader, sending a full tablet report...
Jun 19 02:52:47 dist-test-slave-jt0t krb5kdc[5011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.193@KRBTEST.COM
I20260619 02:52:47.200273 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.191598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:45255 (local address 127.0.0.1:55610)
0619 02:52:47.191797 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:47.191845 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:47.191926 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:47.192308 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:47.192313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:47.192717 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:47.192795 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.192798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.193912 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.193913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.194857 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.194861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.195438 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.195439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.195521 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.196089 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:47.196094 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:47.197669 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:47.199241 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:47.199247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:47.199249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:47.199439 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:47.199616 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:47.199617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:47.199617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:47.199673 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:47.199917 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:47.199923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:47.199975 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 02:52:47.200049 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 02:52:47.200168 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
I20260619 02:52:47.200271 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.191830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.21.164.193:45255)
0619 02:52:47.191975 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.191979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.191989 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:52:47.192023 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.192025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.192042 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.192120 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:47.192888 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.193837 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.194947 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.195358 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.195427 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.197747 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:47.197755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:47.197755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:47.197763 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:47.199168 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.199510 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.199513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.199513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.199564 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.199721 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.199722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.199722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.199859 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:47.199930 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:47.200090 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 02:52:47.200179 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260619 02:52:47.200803 5712 tablet_service.cc:1514] Processing DeleteTablet for tablet d37f5068478a473eb1f4cfde79f15e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:26 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:55610
I20260619 02:52:47.201514 5787 tablet_replica.cc:330] stopping tablet replica
I20260619 02:52:47.201630 5787 raft_consensus.cc:2229] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:47.201705 5787 raft_consensus.cc:2258] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:47.202047 5787 ts_tablet_manager.cc:1905] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:47.204361 5787 ts_tablet_manager.cc:1918] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:47.204427 5787 log.cc:1196] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal/wals/d37f5068478a473eb1f4cfde79f15e30
I20260619 02:52:47.204607 5787 ts_tablet_manager.cc:1939] T d37f5068478a473eb1f4cfde79f15e30 P ef99a8f4f6c6462387cc2e4b63d2e514: Deleting consensus metadata
I20260619 02:52:47.205174 5576 catalog_manager.cc:4841] TS ef99a8f4f6c6462387cc2e4b63d2e514 (127.21.164.193:45255): tablet d37f5068478a473eb1f4cfde79f15e30 (table test-table [id=65806bb3bf874271b6acdc2877a1c292]) successfully deleted
W20260619 02:52:47.275099 5783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:47.275352 5783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:47.275377 5783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:47.276588 5783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:47.276638 5783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:47.276695 5783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:52:47.278195 5783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:36031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=32997
--tserver_master_addrs=127.21.164.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:47.278435 5783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:47.278680 5783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:52:47.284026 5783 server_base.cc:1034] running on GCE node
W20260619 02:52:47.283995 5793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:47.283996 5795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:47.283978 5792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:47.284448 5783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:47.284715 5783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:47.285853 5783 hybrid_clock.cc:648] HybridClock initialized: now 1781837567285842 us; error 31 us; skew 500 ppm
Jun 19 02:52:47 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:47.288533 5783 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:52:47.288997 5783 webserver.cc:460] Webserver started at http://127.21.164.194:32997/ using document root <none> and password file <none>
I20260619 02:52:47.289201 5783 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:47.289268 5783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:47.290654 5783 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:47.291258 5802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:47.291409 5783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:52:47.291489 5783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "a33a3a4f7e4c4ef2b3354648215f8974"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "4aadfd5f575199e2b8e99566dd623ea5"
server_key_iv: "a0b5c7276fe12b483819daf8b3f1361a"
server_key_version: "encryptionkey@0"
I20260619 02:52:47.291843 5783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:47.310809 5783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:47.313212 5783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:47.313372 5783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:47.313654 5783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:47.314165 5809 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:52:47.315819 5783 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:52:47.315872 5783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 02:52:47.315918 5783 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:52:47.316704 5783 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:52:47.316738 5783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:47.316804 5809 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Bootstrap starting.
I20260619 02:52:47.322664 5783 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:36031
I20260619 02:52:47.322759 5916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:36031 every 8 connection(s)
I20260619 02:52:47.323093 5783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:52:47.326396 5809 log.cc:824] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:47.328508 5809 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:52:47.328761 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 5783
I20260619 02:52:47.328868 5809 tablet_bootstrap.cc:492] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Bootstrap complete.
I20260619 02:52:47.329063 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:45423
--local_ip_for_outbound_sockets=127.21.164.195
--tserver_master_addrs=127.21.164.254:34175
--webserver_port=34245
--webserver_interface=127.21.164.195
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:52:47.329880 5809 ts_tablet_manager.cc:1397] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.012s
Jun 19 02:52:47 dist-test-slave-jt0t krb5kdc[5011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:47.331125 5809 raft_consensus.cc:348] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:47.331413 5809 raft_consensus.cc:729] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33a3a4f7e4c4ef2b3354648215f8974, State: Initialized, Role: FOLLOWER
I20260619 02:52:47.331574 5809 consensus_queue.cc:260] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } } peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } }
I20260619 02:52:47.331840 5809 ts_tablet_manager.cc:1428] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 02:52:47.333443 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.323958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:51499 (local address 127.21.164.254:34175)
0619 02:52:47.324048 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.324050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.324861 (+ 811us) server_negotiation.cc:407] Connection header received
0619 02:52:47.325363 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.325365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.325380 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.325434 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:47.326093 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.326689 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.327715 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.327944 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.328002 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.330393 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:47.330394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:47.330395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:47.330402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:47.332065 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.332440 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.332442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.332442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.332504 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.332778 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.332779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.332779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.332900 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:47.332959 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:47.333235 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 02:52:47.333342 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 02:52:47.333557 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.323879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:51499)
0619 02:52:47.324735 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:47.324774 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:47.325262 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:47.325499 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:47.325504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:47.325783 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:47.325996 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.326001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.326791 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.326793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.327631 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.327633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.328040 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.328041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.328167 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.328719 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:47.328728 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:47.330300 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:47.332130 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:47.332137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:47.332139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:47.332372 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:47.332566 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:47.332567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:47.332567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:47.332717 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:47.332954 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:47.332961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:47.333109 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 02:52:47.333193 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 02:52:47.333358 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":81,"threads_started":1}
I20260619 02:52:47.334110 5917 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34175
I20260619 02:52:47.334228 5917 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:47.334431 5917 heartbeater.cc:511] Master 127.21.164.254:34175 requested a full tablet report, sending...
I20260619 02:52:47.335096 5590 ts_manager.cc:194] Registered new tserver with Master: a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031)
I20260619 02:52:47.335669 5590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:51499
I20260619 02:52:47.336503 5917 heartbeater.cc:503] Master 127.21.164.254:34175 was elected leader, sending a full tablet report...
Jun 19 02:52:47 dist-test-slave-jt0t krb5kdc[5011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.194@KRBTEST.COM
I20260619 02:52:47.343254 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.335664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.21.164.194:36031)
0619 02:52:47.335854 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.335860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.335871 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:52:47.335908 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.335910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.335928 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.336021 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:47.336326 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.337419 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.338068 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.338524 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.338601 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.340572 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:47.340579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:47.340579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:47.340586 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:47.342113 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.342503 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.342507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.342507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.342572 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.342744 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.342745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.342745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.342877 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:47.342935 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:47.343064 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 02:52:47.343154 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":103,"threads_started":1}
I20260619 02:52:47.343268 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.335420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:36031 (local address 127.0.0.1:44818)
0619 02:52:47.335498 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:47.335511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:47.335564 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:47.336026 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:47.336027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:47.336200 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:47.336246 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.336249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.337492 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.337493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.337997 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.337999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.338629 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.338630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.338698 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.339149 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:47.339152 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:47.340475 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:47.342188 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:47.342190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:47.342190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:47.342404 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:47.342632 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:47.342633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:47.342633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:47.342693 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:47.342926 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:47.342927 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:47.342974 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 02:52:47.343041 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 02:52:47.343166 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 02:52:47.343751 5851 tablet_service.cc:1514] Processing DeleteTablet for tablet d37f5068478a473eb1f4cfde79f15e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:26 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:44818
I20260619 02:52:47.344300 5924 tablet_replica.cc:330] stopping tablet replica
I20260619 02:52:47.344398 5924 raft_consensus.cc:2229] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:47.344470 5924 raft_consensus.cc:2258] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:47.344806 5924 ts_tablet_manager.cc:1905] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:47.347082 5924 ts_tablet_manager.cc:1918] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:47.347167 5924 log.cc:1196] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/d37f5068478a473eb1f4cfde79f15e30
I20260619 02:52:47.347364 5924 ts_tablet_manager.cc:1939] T d37f5068478a473eb1f4cfde79f15e30 P a33a3a4f7e4c4ef2b3354648215f8974: Deleting consensus metadata
I20260619 02:52:47.347884 5577 catalog_manager.cc:4841] TS a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031): tablet d37f5068478a473eb1f4cfde79f15e30 (table test-table [id=65806bb3bf874271b6acdc2877a1c292]) successfully deleted
W20260619 02:52:47.416777 5921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:52:47.416985 5921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:52:47.417007 5921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:52:47.418120 5921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:52:47.418155 5921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:52:47.418179 5921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:52:47.419579 5921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:45423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=34245
--tserver_master_addrs=127.21.164.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:52:47.419772 5921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:52:47.419958 5921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:52:47.424697 5932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:47.424701 5930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:52:47.424753 5929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:52:47.424801 5921 server_base.cc:1034] running on GCE node
I20260619 02:52:47.425124 5921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:52:47.425351 5921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:52:47.426524 5921 hybrid_clock.cc:648] HybridClock initialized: now 1781837567426511 us; error 28 us; skew 500 ppm
Jun 19 02:52:47 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:52:47.429260 5921 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:52:47.429706 5921 webserver.cc:460] Webserver started at http://127.21.164.195:34245/ using document root <none> and password file <none>
I20260619 02:52:47.429924 5921 fs_manager.cc:359] Metadata directory not provided
I20260619 02:52:47.429986 5921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:52:47.431437 5921 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:47.432121 5939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:47.432278 5921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:47.432374 5921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "e0bba9c3872d4add9b94166ad19fb1d4"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "4c81767070fc632fbd2b30306ed38560"
server_key_iv: "d508a804c714bc74831fa9852ea954a8"
server_key_version: "encryptionkey@0"
I20260619 02:52:47.432760 5921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:52:47.451423 5921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:52:47.453290 5921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:52:47.453433 5921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:52:47.453663 5921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:52:47.453949 5921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:52:47.453984 5921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:47.454012 5921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:52:47.454034 5921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:47.459486 5921 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:45423
I20260619 02:52:47.459587 6052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:45423 every 8 connection(s)
I20260619 02:52:47.459890 5921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:52:47.465072 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 5921
Jun 19 02:52:47 dist-test-slave-jt0t krb5kdc[5011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:52:47.468674 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.460610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:56321 (local address 127.21.164.254:34175)
0619 02:52:47.460704 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.460706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.460885 (+ 179us) server_negotiation.cc:407] Connection header received
0619 02:52:47.461365 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.461366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.461378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.461429 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:47.462369 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.462881 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.463862 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.464067 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.464131 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.466243 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:47.466245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:47.466245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:47.466252 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:47.467464 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.467810 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.467811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.467811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.467863 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.468073 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.468073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.468074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.468178 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:47.468231 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:47.468480 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 02:52:47.468569 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:52:47.468845 6055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.460601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:56321)
0619 02:52:47.460770 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:47.460808 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:47.461277 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:47.461485 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:47.461488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:47.462063 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:52:47.462265 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.462270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.462951 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.462952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.463773 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.463776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.464145 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.464146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.464241 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.464755 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:52:47.464763 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:52:47.466153 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:52:47.467535 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:47.467540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:47.467543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:47.467746 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:47.467915 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:52:47.467916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:52:47.467917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:52:47.468019 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:52:47.468229 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:52:47.468236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:52:47.468367 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 02:52:47.468452 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 02:52:47.468606 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260619 02:52:47.469364 6053 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34175
I20260619 02:52:47.469470 6053 heartbeater.cc:464] Registering TS with master...
I20260619 02:52:47.469658 6053 heartbeater.cc:511] Master 127.21.164.254:34175 requested a full tablet report, sending...
I20260619 02:52:47.470027 5590 ts_manager.cc:194] Registered new tserver with Master: e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423)
I20260619 02:52:47.470387 5590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:56321
I20260619 02:52:47.471442 6056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.465436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.21.164.254:34175)
0619 02:52:47.465614 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.465616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.465675 (+ 59us) server_negotiation.cc:407] Connection header received
0619 02:52:47.465718 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.465719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.465731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.465791 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:47.466349 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.466823 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.467559 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.467759 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.467810 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.468911 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:47.468912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:47.468912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:47.468919 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:47.470389 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.470784 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.470785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.470785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.470846 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.471003 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.471004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.471005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.471125 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:47.471273 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:47.471308 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 02:52:47.471371 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260619 02:52:47.472072 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:52:47.474695 6056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.472388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.21.164.254:34175)
0619 02:52:47.472442 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.472444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.472508 (+ 64us) server_negotiation.cc:407] Connection header received
0619 02:52:47.472572 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.472573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.472606 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.472688 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:47.472801 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.473259 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.473967 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.474219 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.474273 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.474425 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:47.474475 (+ 50us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 02:52:47.474634 (+ 159us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56600: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 02:52:47.474745 6056 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56600: token expired
W20260619 02:52:47.474821 5521 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.164.254:34175)
I20260619 02:52:47.479919 6056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.475037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.21.164.254:34175)
0619 02:52:47.475078 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.475080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.475102 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:52:47.475156 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.475156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.475164 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.475203 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:52:47.475634 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.476083 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.476770 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.476959 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.477011 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.477992 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:52:47.477994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:52:47.477994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:52:47.478001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:52:47.478969 (+ 968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.479260 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.479261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.479261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.479311 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:52:47.479465 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:52:47.479466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:52:47.479466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:52:47.479575 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:52:47.479658 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:47.479752 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 02:52:47.479828 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:52:47.481093 5590 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:52:47.482751 5590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:52:47.484726 5851 tablet_service.cc:1467] Processing CreateTablet for tablet a028a2ab7ae8411abd73da0279c6797d (DEFAULT_TABLE table=test-table [id=463a8af09592426aa385722cbcdfd477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:47.484768 5712 tablet_service.cc:1467] Processing CreateTablet for tablet a028a2ab7ae8411abd73da0279c6797d (DEFAULT_TABLE table=test-table [id=463a8af09592426aa385722cbcdfd477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:47.484920 5712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a028a2ab7ae8411abd73da0279c6797d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:47.484920 5851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a028a2ab7ae8411abd73da0279c6797d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:47.486469 5809 tablet_bootstrap.cc:492] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974: Bootstrap starting.
I20260619 02:52:47.486810 5809 tablet_bootstrap.cc:654] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:47.487068 5670 tablet_bootstrap.cc:492] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: Bootstrap starting.
I20260619 02:52:47.487416 5809 tablet_bootstrap.cc:492] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974: No bootstrap required, opened a new log
I20260619 02:52:47.487459 5809 ts_tablet_manager.cc:1397] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:47.487478 5670 tablet_bootstrap.cc:654] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:47.487798 5809 raft_consensus.cc:348] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:52:47.487872 5809 raft_consensus.cc:374] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:47.487905 5809 raft_consensus.cc:729] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33a3a4f7e4c4ef2b3354648215f8974, State: Initialized, Role: FOLLOWER
I20260619 02:52:47.487950 5809 consensus_queue.cc:260] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:52:47.488035 5809 ts_tablet_manager.cc:1428] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 02:52:47.488049 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.484458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:45423 (local address 127.0.0.1:36142)
0619 02:52:47.484490 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:47.484499 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:47.484531 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:47.485001 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:47.485002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:47.485004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:47.485046 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.485048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.485959 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.485960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.486848 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.486852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.487757 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.487759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.487837 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.487862 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 02:52:47.487928 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:52:47.487975 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 02:52:47.488080 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.484599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.21.164.195:45423)
0619 02:52:47.484831 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.484836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.484848 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:52:47.484889 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.484891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.484907 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.484989 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:47.485100 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.485886 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.486937 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.487660 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.487743 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.487800 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:47.487952 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 02:52:47.487999 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":144,"threads_started":1}
I20260619 02:52:47.488385 5670 tablet_bootstrap.cc:492] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: No bootstrap required, opened a new log
I20260619 02:52:47.488436 5670 ts_tablet_manager.cc:1397] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 02:52:47.488587 5670 raft_consensus.cc:348] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:52:47.488759 5670 raft_consensus.cc:374] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:47.488814 5670 raft_consensus.cc:729] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef99a8f4f6c6462387cc2e4b63d2e514, State: Initialized, Role: FOLLOWER
I20260619 02:52:47.488864 5670 consensus_queue.cc:260] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:52:47.488961 5670 ts_tablet_manager.cc:1428] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 02:52:47.488869 5987 tablet_service.cc:1467] Processing CreateTablet for tablet a028a2ab7ae8411abd73da0279c6797d (DEFAULT_TABLE table=test-table [id=463a8af09592426aa385722cbcdfd477]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:52:47.489217 5987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a028a2ab7ae8411abd73da0279c6797d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:52:47.492188 6062 tablet_bootstrap.cc:492] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: Bootstrap starting.
I20260619 02:52:47.492795 6062 tablet_bootstrap.cc:654] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: Neither blocks nor log segments found. Creating new log.
I20260619 02:52:47.493235 6062 log.cc:824] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:52:47.493886 6062 tablet_bootstrap.cc:492] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: No bootstrap required, opened a new log
I20260619 02:52:47.493943 6062 ts_tablet_manager.cc:1397] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:52:47.495400 6062 raft_consensus.cc:348] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:52:47.495540 6062 raft_consensus.cc:374] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:52:47.495561 6062 raft_consensus.cc:729] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0bba9c3872d4add9b94166ad19fb1d4, State: Initialized, Role: FOLLOWER
I20260619 02:52:47.495658 6062 consensus_queue.cc:260] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:52:47.495898 6062 ts_tablet_manager.cc:1428] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:52:47.495930 6053 heartbeater.cc:503] Master 127.21.164.254:34175 was elected leader, sending a full tablet report...
W20260619 02:52:47.573696 5918 tablet.cc:2369] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:47.664896 5922 raft_consensus.cc:482] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:52:47.665081 5922 raft_consensus.cc:504] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:52:47.665427 5922 leader_election.cc:288] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423), ef99a8f4f6c6462387cc2e4b63d2e514 (127.21.164.193:45255)
I20260619 02:52:47.669075 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.665665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:37841 (local address 127.21.164.195:45423)
0619 02:52:47.665714 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.665716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.665727 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:52:47.665815 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.665816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.665827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.665893 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:47.666074 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.666678 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.668095 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.668692 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.668776 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.668786 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:47.668959 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 02:52:47.669000 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:52:47.669075 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.665678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:44351 (local address 127.21.164.193:45255)
0619 02:52:47.665721 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.665723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.665948 (+ 225us) server_negotiation.cc:407] Connection header received
0619 02:52:47.665995 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.665996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.666024 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.666092 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:52:47.666243 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.666851 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.668100 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.668746 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.668806 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.668889 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:47.668959 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 02:52:47.669001 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:52:47.669085 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.665605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:45423 (local address 127.21.164.194:37841)
0619 02:52:47.665670 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:47.665683 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:47.665737 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:47.665913 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:47.665913 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:47.665915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:47.665991 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.665993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.666777 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.666778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.667990 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.667993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.668773 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.668774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.668856 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.668887 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:52:47.668946 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:52:47.668991 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":3}
I20260619 02:52:47.669108 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.665606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:45255 (local address 127.21.164.194:44351)
0619 02:52:47.665871 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:47.665882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:52:47.665938 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:52:47.666105 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:52:47.666106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:52:47.666108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:52:47.666180 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.666182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.666912 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.666913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.667990 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:52:47.667993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.668823 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.668824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:52:47.668879 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.668890 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 02:52:47.668946 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:52:47.668991 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":92,"thread_start_us":109,"threads_started":1}
I20260619 02:52:47.669425 6007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a028a2ab7ae8411abd73da0279c6797d" candidate_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" is_pre_election: true
I20260619 02:52:47.669608 6007 raft_consensus.cc:2454] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a33a3a4f7e4c4ef2b3354648215f8974 in term 0.
I20260619 02:52:47.669831 5804 leader_election.cc:302] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a33a3a4f7e4c4ef2b3354648215f8974, e0bba9c3872d4add9b94166ad19fb1d4; no voters:
I20260619 02:52:47.669972 5922 raft_consensus.cc:2781] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:52:47.669945 5732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a028a2ab7ae8411abd73da0279c6797d" candidate_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" is_pre_election: true
I20260619 02:52:47.670023 5922 raft_consensus.cc:482] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:52:47.670046 5922 raft_consensus.cc:3037] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:47.670096 5732 raft_consensus.cc:2454] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a33a3a4f7e4c4ef2b3354648215f8974 in term 0.
I20260619 02:52:47.670835 5922 raft_consensus.cc:504] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:52:47.670955 5922 leader_election.cc:288] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [CANDIDATE]: Term 1 election: Requested vote from peers e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423), ef99a8f4f6c6462387cc2e4b63d2e514 (127.21.164.193:45255)
I20260619 02:52:47.671099 5732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a028a2ab7ae8411abd73da0279c6797d" candidate_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514"
I20260619 02:52:47.671144 6007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a028a2ab7ae8411abd73da0279c6797d" candidate_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bba9c3872d4add9b94166ad19fb1d4"
I20260619 02:52:47.671177 5732 raft_consensus.cc:3037] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:47.671208 6007 raft_consensus.cc:3037] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:52:47.671844 5732 raft_consensus.cc:2454] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a33a3a4f7e4c4ef2b3354648215f8974 in term 1.
I20260619 02:52:47.671893 6007 raft_consensus.cc:2454] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a33a3a4f7e4c4ef2b3354648215f8974 in term 1.
I20260619 02:52:47.672014 5804 leader_election.cc:302] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a33a3a4f7e4c4ef2b3354648215f8974, e0bba9c3872d4add9b94166ad19fb1d4; no voters:
I20260619 02:52:47.672104 5922 raft_consensus.cc:2781] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:52:47.672240 5922 raft_consensus.cc:686] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 LEADER]: Becoming Leader. State: Replica: a33a3a4f7e4c4ef2b3354648215f8974, State: Running, Role: LEADER
I20260619 02:52:47.672295 5922 consensus_queue.cc:237] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:52:47.673014 5590 catalog_manager.cc:5495] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 reported cstate change: term changed from 0 to 1, leader changed from <none> to a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194). New cstate: current_term: 1 leader_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } health_report { overall_health: UNKNOWN } } }
W20260619 02:52:47.677232 5779 tablet.cc:2369] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:52:47.710433 6054 tablet.cc:2369] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:52:47.716740 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.714538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.21.164.194:36031)
0619 02:52:47.714590 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 02:52:47.714592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:52:47.714599 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:52:47.714638 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:52:47.714639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:52:47.714655 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:52:47.714710 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:52:47.714815 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.715298 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.716011 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:52:47.716208 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:52:47.716253 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:52:47.716394 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:52:47.716491 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:52:47.716540 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 02:52:47.716629 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 02:52:47.716670 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:52:47.718223 5732 raft_consensus.cc:1261] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Refusing update from remote peer a33a3a4f7e4c4ef2b3354648215f8974: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:47.718231 6007 raft_consensus.cc:1261] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4 [term 1 FOLLOWER]: Refusing update from remote peer a33a3a4f7e4c4ef2b3354648215f8974: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:52:47.718583 6066 consensus_queue.cc:1035] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:47.718705 5922 consensus_queue.cc:1035] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:52:47.720453 6069 mvcc.cc:205] Tried to move back new op lower bound from 7298406677371617280 to 7298406677186269184. Current Snapshot: MvccSnapshot[applied={T|T < 7298406677371617280}]
I20260619 02:52:47.721159 6070 mvcc.cc:205] Tried to move back new op lower bound from 7298406677371617280 to 7298406677186269184. Current Snapshot: MvccSnapshot[applied={T|T < 7298406677371617280 or (T in {7298406677371617280})}]
I20260619 02:52:47.722739 5590 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:47.722828 5590 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:52:47.723521 5590 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Sending DeleteTablet for 3 replicas of tablet a028a2ab7ae8411abd73da0279c6797d
I20260619 02:52:47.723819 5712 tablet_service.cc:1514] Processing DeleteTablet for tablet a028a2ab7ae8411abd73da0279c6797d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:47 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:55610
I20260619 02:52:47.723954 5851 tablet_service.cc:1514] Processing DeleteTablet for tablet a028a2ab7ae8411abd73da0279c6797d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:47 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:44818
I20260619 02:52:47.723989 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 5628
I20260619 02:52:47.724026 6079 tablet_replica.cc:330] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: stopping tablet replica
I20260619 02:52:47.724043 5987 tablet_service.cc:1514] Processing DeleteTablet for tablet a028a2ab7ae8411abd73da0279c6797d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:52:47 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:36142
I20260619 02:52:47.724082 6079 raft_consensus.cc:2229] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:47.724128 6079 raft_consensus.cc:2258] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:47.724221 5924 tablet_replica.cc:330] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974: stopping tablet replica
I20260619 02:52:47.724264 5924 raft_consensus.cc:2229] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:52:47.724336 5924 raft_consensus.cc:2258] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:47.724562 6080 tablet_replica.cc:330] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: stopping tablet replica
I20260619 02:52:47.724730 6080 raft_consensus.cc:2229] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:52:47.724838 6080 raft_consensus.cc:2258] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:52:47.725292 6080 ts_tablet_manager.cc:1905] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:47.725904 5924 ts_tablet_manager.cc:1905] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:52:47.727276 5924 ts_tablet_manager.cc:1918] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:47.727337 5924 log.cc:1196] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/a028a2ab7ae8411abd73da0279c6797d
I20260619 02:52:47.727526 5924 ts_tablet_manager.cc:1939] T a028a2ab7ae8411abd73da0279c6797d P a33a3a4f7e4c4ef2b3354648215f8974: Deleting consensus metadata
I20260619 02:52:47.727978 5577 catalog_manager.cc:4841] TS a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031): tablet a028a2ab7ae8411abd73da0279c6797d (table test-table [id=463a8af09592426aa385722cbcdfd477]) successfully deleted
W20260619 02:52:47.730091 5576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:45255 (error 108)
I20260619 02:52:47.730345 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 5783
I20260619 02:52:47.730386 6080 ts_tablet_manager.cc:1918] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:52:47.730476 6080 log.cc:1196] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/a028a2ab7ae8411abd73da0279c6797d
I20260619 02:52:47.730791 6080 ts_tablet_manager.cc:1939] T a028a2ab7ae8411abd73da0279c6797d P e0bba9c3872d4add9b94166ad19fb1d4: Deleting consensus metadata
I20260619 02:52:47.731272 5575 catalog_manager.cc:4841] TS e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423): tablet a028a2ab7ae8411abd73da0279c6797d (table test-table [id=463a8af09592426aa385722cbcdfd477]) successfully deleted
I20260619 02:52:47.734642 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 5921
I20260619 02:52:47.735414 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:52:47.734794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:45255 (local address 127.0.0.1:55618)
0619 02:52:47.735331 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 02:52:47.735380 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:45255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516}
W20260619 02:52:47.735575 5576 catalog_manager.cc:4584] TS ef99a8f4f6c6462387cc2e4b63d2e514 (127.21.164.193:45255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a028a2ab7ae8411abd73da0279c6797d: Network error: Client connection negotiation failed: client connection to 127.21.164.193:45255: connect: Connection refused (error 111)
I20260619 02:52:47.739746 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 5557
I20260619 02:53:07.744565 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34175
--webserver_interface=127.21.164.254
--webserver_port=42673
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:34175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:07.830120 6082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:07.830360 6082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:07.830389 6082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:07.831530 6082 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 02:53:07.831570 6082 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 02:53:07.831586 6082 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 02:53:07.831599 6082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:07.831615 6082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:07.831633 6082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:07.831653 6082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:07.831676 6082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:07.833244 6082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:34175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=42673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:07.833513 6082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:07.833724 6082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:07.838603 6090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:07.838610 6088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:07.838661 6087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:07.838646 6082 server_base.cc:1034] running on GCE node
I20260619 02:53:07.838989 6082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:07.839287 6082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:07.840420 6082 hybrid_clock.cc:648] HybridClock initialized: now 1781837587840409 us; error 31 us; skew 500 ppm
Jun 19 02:53:07 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:07.847708 6082 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:07.848176 6082 webserver.cc:460] Webserver started at http://127.21.164.254:42673/ using document root <none> and password file <none>
I20260619 02:53:07.848372 6082 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:07.848416 6082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:07.849887 6082 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:07.850613 6097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:07.850817 6082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:07.850907 6082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "78e79060646d4f7dbcf1134384fb9579"
format_stamp: "Formatted at 2026-06-19 02:52:25 on dist-test-slave-jt0t"
server_key: "3c5722996045788975fba565f9c585e7"
server_key_iv: "222b07f23250e0748a5b72c6aa675c46"
server_key_version: "encryptionkey@0"
I20260619 02:53:07.851356 6082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:07.870390 6082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:07.872346 6082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:07.872463 6082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:07.876351 6082 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:34175
I20260619 02:53:07.876396 6149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:34175 every 8 connection(s)
I20260619 02:53:07.876776 6082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:07.878245 6150 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:53:07.878923 6150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Bootstrap starting.
I20260619 02:53:07.880594 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 6082
I20260619 02:53:07.881028 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:45255
--local_ip_for_outbound_sockets=127.21.164.193
--tserver_master_addrs=127.21.164.254:34175
--webserver_port=38043
--webserver_interface=127.21.164.193
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:53:07.887753 6150 log.cc:824] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:07.890079 6150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:53:07.890412 6150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Bootstrap complete.
I20260619 02:53:07.892481 6150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:53:07.892736 6150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e79060646d4f7dbcf1134384fb9579, State: Initialized, Role: FOLLOWER
I20260619 02:53:07.892897 6150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:53:07.892952 6150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:07.892995 6150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:07.893040 6150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:53:07.894016 6150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:53:07.894104 6150 leader_election.cc:302] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78e79060646d4f7dbcf1134384fb9579; no voters:
I20260619 02:53:07.894258 6150 leader_election.cc:288] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 02:53:07.894309 6154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:53:07.894465 6154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [term 3 LEADER]: Becoming Leader. State: Replica: 78e79060646d4f7dbcf1134384fb9579, State: Running, Role: LEADER
I20260619 02:53:07.894534 6150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:07.894639 6154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } }
I20260619 02:53:07.894975 6155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "78e79060646d4f7dbcf1134384fb9579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } } }
I20260619 02:53:07.894975 6154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78e79060646d4f7dbcf1134384fb9579. Latest consensus state: current_term: 3 leader_uuid: "78e79060646d4f7dbcf1134384fb9579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e79060646d4f7dbcf1134384fb9579" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34175 } } }
I20260619 02:53:07.895167 6154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:07.895387 6155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:07.895388 6163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:07.896413 6163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:07.896507 6163 catalog_manager.cc:1259] Loaded cluster ID: 655b200cc9894ae9a13cf4bfd39d4651
I20260619 02:53:07.896538 6163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:07.897372 6163 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:07.897543 6163 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Loaded TSK: 0
I20260619 02:53:07.897578 6163 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 78e79060646d4f7dbcf1134384fb9579: Loaded TSK: 1
F20260619 02:53:07.897792 6163 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f457af929ad google::LogMessage::Fail() at ??:0
@ 0x7f457af9588a google::LogMessage::SendToLog() at ??:0
@ 0x7f457af924de google::LogMessage::Flush() at ??:0
@ 0x7f457af93a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f457d6ff104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f457d6fcd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4582eec1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4582eec48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4582eec594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4582eb2c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f4582ebcad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f457ce600d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f457ce55954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f457dcb96db start_thread at ??:0
@ 0x7f4579a9061f clone at ??:0
W20260619 02:53:07.978830 6152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:07.979053 6152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:07.979089 6152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:07.980381 6152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:07.980424 6152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:07.980463 6152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:53:07.982009 6152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:45255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=38043
--tserver_master_addrs=127.21.164.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:07.982239 6152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:07.982461 6152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:53:07.987994 6152 server_base.cc:1034] running on GCE node
W20260619 02:53:07.988092 6192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:07.987968 6191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:07.987978 6194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:07.988498 6152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:07.988776 6152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:07.989933 6152 hybrid_clock.cc:648] HybridClock initialized: now 1781837587989919 us; error 36 us; skew 500 ppm
Jun 19 02:53:07 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:07.992601 6152 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:53:07.993060 6152 webserver.cc:460] Webserver started at http://127.21.164.193:38043/ using document root <none> and password file <none>
I20260619 02:53:07.993286 6152 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:07.993335 6152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:07.994676 6152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:53:07.995340 6201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:07.995512 6152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 02:53:07.995604 6152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "ef99a8f4f6c6462387cc2e4b63d2e514"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "8aa086c95d08d5bb86179348b00ce95b"
server_key_iv: "440bd6d86a0bace099a2e133e7b00853"
server_key_version: "encryptionkey@0"
I20260619 02:53:07.996026 6152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:08.010381 6152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:08.012997 6152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:08.013160 6152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:08.013453 6152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:08.014051 6208 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 02:53:08.015498 6152 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 02:53:08.015558 6152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 02:53:08.015590 6152 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 02:53:08.016465 6152 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 02:53:08.016511 6152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 02:53:08.016605 6208 tablet_bootstrap.cc:492] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: Bootstrap starting.
I20260619 02:53:08.023522 6152 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:45255
I20260619 02:53:08.023712 6315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:45255 every 8 connection(s)
I20260619 02:53:08.024040 6152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:08.026439 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 6152
I20260619 02:53:08.026806 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:36031
--local_ip_for_outbound_sockets=127.21.164.194
--tserver_master_addrs=127.21.164.254:34175
--webserver_port=32997
--webserver_interface=127.21.164.194
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:53:08.026868 6208 log.cc:824] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:08.029699 6208 tablet_bootstrap.cc:492] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:53:08.030014 6208 tablet_bootstrap.cc:492] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: Bootstrap complete.
I20260619 02:53:08.031105 6208 ts_tablet_manager.cc:1397] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260619 02:53:08.032527 6208 raft_consensus.cc:348] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:53:08.032881 6208 raft_consensus.cc:729] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef99a8f4f6c6462387cc2e4b63d2e514, State: Initialized, Role: FOLLOWER
I20260619 02:53:08.033056 6208 consensus_queue.cc:260] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:53:08.033339 6208 ts_tablet_manager.cc:1428] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260619 02:53:08.095002 6318 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:08.024942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:56587)
0619 02:53:08.025162 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:08.025224 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:08.025937 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:08.094800 (+ 68863us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":104,"threads_started":1}
W20260619 02:53:08.095250 6316 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:34175 (0 consecutive failures): Network error: Failed to ping master at 127.21.164.254:34175: Client connection negotiation failed: client connection to 127.21.164.254:34175: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:53:08.095597 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:08.095476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:57695)
0619 02:53:08.095526 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:08.095571 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 02:53:08.095911 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:08.095820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:38915)
0619 02:53:08.095862 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:08.095890 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 02:53:08.096019 6316 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:53:08.109664 6320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:08.109889 6320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:08.109913 6320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:08.111135 6320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:08.111186 6320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:08.111214 6320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:53:08.112749 6320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:36031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=32997
--tserver_master_addrs=127.21.164.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:08.112944 6320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:08.113128 6320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:08.118041 6326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:08.118165 6320 server_base.cc:1034] running on GCE node
W20260619 02:53:08.118083 6329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:08.118355 6327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:08.118603 6320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:08.118933 6320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:08.120105 6320 hybrid_clock.cc:648] HybridClock initialized: now 1781837588120084 us; error 38 us; skew 500 ppm
Jun 19 02:53:08 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837588, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:08.123277 6320 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:53:08.123807 6320 webserver.cc:460] Webserver started at http://127.21.164.194:32997/ using document root <none> and password file <none>
I20260619 02:53:08.124097 6320 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:08.124159 6320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:08.125723 6320 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:08.126407 6336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:08.126559 6320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:08.126652 6320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "a33a3a4f7e4c4ef2b3354648215f8974"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "4aadfd5f575199e2b8e99566dd623ea5"
server_key_iv: "a0b5c7276fe12b483819daf8b3f1361a"
server_key_version: "encryptionkey@0"
I20260619 02:53:08.127094 6320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:08.137317 6320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:08.139776 6320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:08.139942 6320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:08.140189 6320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:08.140574 6320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:08.140621 6320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:08.140684 6320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:08.140712 6320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:08.147786 6320 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:36031
I20260619 02:53:08.147845 6449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:36031 every 8 connection(s)
I20260619 02:53:08.148238 6320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:08.149552 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:08.148982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:38423)
0619 02:53:08.149140 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:08.149210 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
W20260619 02:53:08.149792 6450 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:34175 (0 consecutive failures): Network error: Failed to ping master at 127.21.164.254:34175: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
I20260619 02:53:08.150053 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:08.149960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:57531)
0619 02:53:08.149999 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:08.150027 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260619 02:53:08.150388 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:08.150305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:46247)
0619 02:53:08.150337 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:08.150363 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260619 02:53:08.150481 6450 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:53:08.152891 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 6320
I20260619 02:53:08.153146 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:45423
--local_ip_for_outbound_sockets=127.21.164.195
--tserver_master_addrs=127.21.164.254:34175
--webserver_port=34245
--webserver_interface=127.21.164.195
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:08.237121 6453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:08.237327 6453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:08.237358 6453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:08.238572 6453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:08.238622 6453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:08.238672 6453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:53:08.240109 6453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:45423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=34245
--tserver_master_addrs=127.21.164.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:08.240361 6453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:08.240613 6453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:08.245604 6458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:08.245604 6459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:08.245621 6461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:08.245853 6453 server_base.cc:1034] running on GCE node
I20260619 02:53:08.246052 6453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:08.246277 6453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:08.247417 6453 hybrid_clock.cc:648] HybridClock initialized: now 1781837588247401 us; error 27 us; skew 500 ppm
Jun 19 02:53:08 dist-test-slave-jt0t krb5kdc[5011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837588, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:08.249884 6453 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:53:08.250332 6453 webserver.cc:460] Webserver started at http://127.21.164.195:34245/ using document root <none> and password file <none>
I20260619 02:53:08.250553 6453 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:08.250602 6453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:08.251855 6453 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:53:08.252413 6468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:08.252573 6453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:53:08.252676 6453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "e0bba9c3872d4add9b94166ad19fb1d4"
format_stamp: "Formatted at 2026-06-19 02:52:26 on dist-test-slave-jt0t"
server_key: "4c81767070fc632fbd2b30306ed38560"
server_key_iv: "d508a804c714bc74831fa9852ea954a8"
server_key_version: "encryptionkey@0"
I20260619 02:53:08.253036 6453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:08.266023 6453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:08.267928 6453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:08.268067 6453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:08.268289 6453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:08.268581 6453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:08.268613 6453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:08.268643 6453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:08.268687 6453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:08.274120 6453 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:45423
I20260619 02:53:08.274168 6581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:45423 every 8 connection(s)
I20260619 02:53:08.274507 6453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:53:08.275739 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:08.275251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:45261)
0619 02:53:08.275431 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:08.275534 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
W20260619 02:53:08.276223 6582 heartbeater.cc:644] Failed to heartbeat to 127.21.164.254:34175 (0 consecutive failures): Network error: Failed to ping master at 127.21.164.254:34175: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
I20260619 02:53:08.276532 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:08.276441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:52959)
0619 02:53:08.276485 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:08.276513 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 02:53:08.276767 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:08.276713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:58129)
0619 02:53:08.276736 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:08.276754 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260619 02:53:08.276885 6582 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:53:08.278319 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 6453
I20260619 02:53:09.096611 6586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:09.096383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:56535)
0619 02:53:09.096502 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:09.096543 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260619 02:53:09.151034 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:09.150812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:57851)
0619 02:53:09.150938 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:09.150981 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":54,"threads_started":1}
I20260619 02:53:09.277403 6588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:09.277186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:50287)
0619 02:53:09.277296 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:09.277337 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260619 02:53:09.656498 6589 raft_consensus.cc:482] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:09.656620 6589 raft_consensus.cc:504] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:53:09.656989 6589 leader_election.cc:288] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423), a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031)
Jun 19 02:53:09 dist-test-slave-jt0t krb5kdc[5011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.194@KRBTEST.COM
Jun 19 02:53:09 dist-test-slave-jt0t krb5kdc[5011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.195@KRBTEST.COM
I20260619 02:53:09.667609 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:09.657154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:45423 (local address 127.21.164.193:58825)
0619 02:53:09.657274 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:09.657283 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:09.657339 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:09.658248 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:09.658251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:09.658646 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:09.658836 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:09.658841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:09.659952 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:09.659953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:09.660822 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:09.660824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:09.661396 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:09.661396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:09.661508 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:09.662081 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:09.662090 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:09.664061 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:09.665942 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:09.665943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:09.665943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:09.666123 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:09.666377 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:09.666378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:09.666378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:09.666481 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:09.666957 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:09.666964 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:09.667146 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 02:53:09.667267 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 02:53:09.667459 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":100,"thread_start_us":61,"threads_started":1}
I20260619 02:53:09.667622 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:09.657317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:34147 (local address 127.21.164.194:36031)
0619 02:53:09.657463 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:09.657472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:09.657490 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:53:09.658007 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:09.658010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:09.658058 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:09.658191 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:09.658951 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:09.659836 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:09.660895 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:09.661311 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:09.661410 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:09.663846 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:09.663851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:09.663851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:09.663864 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:09.665670 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:09.666148 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:09.666151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:09.666151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:09.666274 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:09.666543 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:09.666543 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:09.666544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:09.666886 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:09.666973 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:09.667357 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 02:53:09.667502 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":46,"threads_started":1}
I20260619 02:53:09.667687 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:09.657154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:36031 (local address 127.21.164.193:34147)
0619 02:53:09.657271 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:09.657283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:09.657339 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:09.658248 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:09.658251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:09.658647 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:09.658836 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:09.658841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:09.659940 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:09.659941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:09.660821 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:09.660823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:09.661398 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:09.661399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:09.661507 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:09.662085 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:09.662094 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:09.663716 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:09.665766 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:09.665779 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:09.665781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:09.666062 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:09.666340 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:09.666341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:09.666341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:09.666481 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:09.666957 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:09.666964 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:09.667146 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 02:53:09.667282 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 02:53:09.667520 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":134,"thread_start_us":46,"threads_started":1}
I20260619 02:53:09.667622 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:09.657238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:58825 (local address 127.21.164.195:45423)
0619 02:53:09.657415 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:09.657428 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:09.657461 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:53:09.658014 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:09.658017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:09.658059 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:09.658202 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:09.658955 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:09.659864 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:09.660906 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:09.661313 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:09.661410 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:09.664163 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:09.664168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:09.664168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:09.664179 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:09.665867 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:09.666190 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:09.666192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:09.666193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:09.666320 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:09.666542 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:09.666543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:09.666543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:09.666887 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:09.666973 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:09.667357 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 02:53:09.667500 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
W20260619 02:53:09.668452 6205 leader_election.cc:341] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031): Not found: Tablet not found: a028a2ab7ae8411abd73da0279c6797d
W20260619 02:53:09.668558 6203 leader_election.cc:341] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423): Not found: Tablet not found: a028a2ab7ae8411abd73da0279c6797d
I20260619 02:53:09.668614 6203 leader_election.cc:302] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef99a8f4f6c6462387cc2e4b63d2e514; no voters: a33a3a4f7e4c4ef2b3354648215f8974, e0bba9c3872d4add9b94166ad19fb1d4
I20260619 02:53:09.668752 6589 raft_consensus.cc:2726] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:53:10.097332 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:10.097174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:51777)
0619 02:53:10.097239 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:10.097278 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 02:53:10.151738 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:10.151482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:47095)
0619 02:53:10.151634 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:10.151685 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260619 02:53:10.278088 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:10.277840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:41633)
0619 02:53:10.277994 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:10.278039 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260619 02:53:11.098017 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:11.097804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:39355)
0619 02:53:11.097929 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:11.097959 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260619 02:53:11.152436 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:11.152178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:43161)
0619 02:53:11.152340 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:11.152382 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260619 02:53:11.216001 6598 raft_consensus.cc:482] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:11.216097 6598 raft_consensus.cc:504] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:53:11.216235 6598 leader_election.cc:288] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423), a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031)
W20260619 02:53:11.216681 6203 leader_election.cc:341] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423): Not found: Tablet not found: a028a2ab7ae8411abd73da0279c6797d
W20260619 02:53:11.216825 6205 leader_election.cc:341] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031): Not found: Tablet not found: a028a2ab7ae8411abd73da0279c6797d
I20260619 02:53:11.216903 6205 leader_election.cc:302] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef99a8f4f6c6462387cc2e4b63d2e514; no voters: a33a3a4f7e4c4ef2b3354648215f8974, e0bba9c3872d4add9b94166ad19fb1d4
I20260619 02:53:11.216977 6598 raft_consensus.cc:2726] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:53:11.278826 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:11.278587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:43575)
0619 02:53:11.278737 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:11.278778 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260619 02:53:12.098791 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:12.098546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:35003)
0619 02:53:12.098694 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:12.098745 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260619 02:53:12.153038 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:12.152853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:42497)
0619 02:53:12.152962 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:12.152993 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260619 02:53:12.279521 6602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:12.279295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:42461)
0619 02:53:12.279441 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:12.279475 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260619 02:53:13.075390 6603 raft_consensus.cc:482] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:13.075498 6603 raft_consensus.cc:504] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:53:13.075680 6603 leader_election.cc:288] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423), a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031)
W20260619 02:53:13.076050 6203 leader_election.cc:341] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423): Not found: Tablet not found: a028a2ab7ae8411abd73da0279c6797d
W20260619 02:53:13.076205 6205 leader_election.cc:341] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031): Not found: Tablet not found: a028a2ab7ae8411abd73da0279c6797d
I20260619 02:53:13.076263 6205 leader_election.cc:302] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef99a8f4f6c6462387cc2e4b63d2e514; no voters: a33a3a4f7e4c4ef2b3354648215f8974, e0bba9c3872d4add9b94166ad19fb1d4
I20260619 02:53:13.076357 6603 raft_consensus.cc:2726] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:53:13.099597 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:13.099344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:55121)
0619 02:53:13.099484 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:13.099537 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20260619 02:53:13.153636 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:13.153433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:54329)
0619 02:53:13.153552 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:13.153588 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260619 02:53:13.280153 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:13.279941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:57951)
0619 02:53:13.280066 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:13.280105 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260619 02:53:14.100310 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:14.100080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:34087)
0619 02:53:14.100214 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:14.100263 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260619 02:53:14.154284 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:14.154067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:41671)
0619 02:53:14.154200 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:14.154237 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260619 02:53:14.280758 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:14.280539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:56473)
0619 02:53:14.280669 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:14.280706 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260619 02:53:15.101145 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:15.100845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:38781)
0619 02:53:15.101027 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:15.101077 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
I20260619 02:53:15.154959 6611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:15.154711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:49589)
0619 02:53:15.154860 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:15.154900 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260619 02:53:15.281386 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:15.281165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:34019)
0619 02:53:15.281300 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:15.281335 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260619 02:53:16.101897 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:16.101635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:45311)
0619 02:53:16.101788 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:16.101832 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260619 02:53:16.155692 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:16.155432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:47295)
0619 02:53:16.155595 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:16.155636 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260619 02:53:16.282112 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:16.281845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:58999)
0619 02:53:16.282011 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:16.282054 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260619 02:53:17.102675 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:17.102428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:49697)
0619 02:53:17.102572 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:17.102622 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260619 02:53:17.156477 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:17.156200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:33961)
0619 02:53:17.156356 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:17.156421 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260619 02:53:17.282886 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:17.282631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:48305)
0619 02:53:17.282788 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:17.282835 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260619 02:53:17.766635 6619 raft_consensus.cc:482] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:17.766736 6619 raft_consensus.cc:504] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0bba9c3872d4add9b94166ad19fb1d4" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45423 } } peers { permanent_uuid: "a33a3a4f7e4c4ef2b3354648215f8974" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 36031 } } peers { permanent_uuid: "ef99a8f4f6c6462387cc2e4b63d2e514" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 45255 } }
I20260619 02:53:17.766882 6619 leader_election.cc:288] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423), a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031)
W20260619 02:53:17.767232 6203 leader_election.cc:341] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0bba9c3872d4add9b94166ad19fb1d4 (127.21.164.195:45423): Not found: Tablet not found: a028a2ab7ae8411abd73da0279c6797d
W20260619 02:53:17.767314 6205 leader_election.cc:341] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a33a3a4f7e4c4ef2b3354648215f8974 (127.21.164.194:36031): Not found: Tablet not found: a028a2ab7ae8411abd73da0279c6797d
I20260619 02:53:17.767361 6205 leader_election.cc:302] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef99a8f4f6c6462387cc2e4b63d2e514; no voters: a33a3a4f7e4c4ef2b3354648215f8974, e0bba9c3872d4add9b94166ad19fb1d4
I20260619 02:53:17.767424 6619 raft_consensus.cc:2726] T a028a2ab7ae8411abd73da0279c6797d P ef99a8f4f6c6462387cc2e4b63d2e514 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 02:53:18.103386 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:18.103180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:32875)
0619 02:53:18.103303 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:18.103338 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260619 02:53:18.157251 6621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:18.156946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:46567)
0619 02:53:18.157127 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:18.157175 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260619 02:53:18.283571 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:18.283343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:55751)
0619 02:53:18.283469 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:18.283513 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260619 02:53:19.103991 6623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:19.103775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:52043)
0619 02:53:19.103912 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:19.103945 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260619 02:53:19.157923 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:19.157686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:56101)
0619 02:53:19.157835 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:19.157873 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260619 02:53:19.284278 6625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:19.284026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:60049)
0619 02:53:19.284179 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:19.284215 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260619 02:53:20.104638 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:20.104435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:33011)
0619 02:53:20.104547 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.104589 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260619 02:53:20.158687 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:20.158440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:58827)
0619 02:53:20.158601 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.158640 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260619 02:53:20.284941 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:20.284745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:39191)
0619 02:53:20.284857 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:20.284893 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260619 02:53:21.105360 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:21.105117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:49557)
0619 02:53:21.105263 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:21.105311 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260619 02:53:21.159415 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:21.159193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:49279)
0619 02:53:21.159332 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:21.159367 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260619 02:53:21.285578 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:21.285363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:38923)
0619 02:53:21.285501 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:21.285532 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260619 02:53:22.106055 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:22.105837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:58297)
0619 02:53:22.105962 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:22.106006 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260619 02:53:22.160120 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:22.159906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:34829)
0619 02:53:22.160040 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:22.160073 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260619 02:53:22.286237 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:22.285987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:60621)
0619 02:53:22.286153 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:22.286191 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":63,"threads_started":1}
I20260619 02:53:23.106812 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:23.106603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.193:53953)
0619 02:53:23.106734 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:23.106768 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260619 02:53:23.160926 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:23.160718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.194:36119)
0619 02:53:23.160837 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:23.160879 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260619 02:53:23.282362 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 6152
I20260619 02:53:23.286324 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 6320
I20260619 02:53:23.287715 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:23.286965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34175 (local address 127.21.164.195:38515)
0619 02:53:23.287624 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:23.287664 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.254:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":51,"threads_started":1}
I20260619 02:53:23.290184 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 6453
2026-06-19T02:53:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58549 ms)
[----------] 30 tests from SecurityITest (227367 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:23 dist-test-slave-jt0t krb5kdc[6640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:23 dist-test-slave-jt0t krb5kdc[6640](info): set up 2 sockets
Jun 19 02:53:23 dist-test-slave-jt0t krb5kdc[6640](info): commencing operation
krb5kdc: starting...
W20260619 02:53:24.252218 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:24 dist-test-slave-jt0t krb5kdc[6640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:53:24.365314 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:43465
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:43465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:24.444432 6653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:24.444684 6653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:24.444717 6653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:24.445885 6653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:24.445917 6653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:24.445931 6653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:24.445942 6653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:24.445952 6653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:24.447402 6653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:43465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:43465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:24.447602 6653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:24.447783 6653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:53:24.452745 6653 server_base.cc:1034] running on GCE node
W20260619 02:53:24.452708 6661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:24.452697 6659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:24.452756 6658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:24.453178 6653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:24.453491 6653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:24.454672 6653 hybrid_clock.cc:648] HybridClock initialized: now 1781837604454650 us; error 32 us; skew 500 ppm
Jun 19 02:53:24 dist-test-slave-jt0t krb5kdc[6640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837604, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:24.457237 6653 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:24.457712 6653 webserver.cc:460] Webserver started at http://127.21.164.254:44683/ using document root <none> and password file <none>
I20260619 02:53:24.457916 6653 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:24.457964 6653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:24.458086 6653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:24.459143 6653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "ec24d75f0c514de88d58e10088a1d653"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "bb967139943737e25cc33ff21c2a3218"
server_key_iv: "857202c43a4f7acac4394825ee32a437"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.459470 6653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "ec24d75f0c514de88d58e10088a1d653"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "bb967139943737e25cc33ff21c2a3218"
server_key_iv: "857202c43a4f7acac4394825ee32a437"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.460743 6653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 02:53:24.461474 6668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:24.461645 6653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:24.461725 6653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "ec24d75f0c514de88d58e10088a1d653"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "bb967139943737e25cc33ff21c2a3218"
server_key_iv: "857202c43a4f7acac4394825ee32a437"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.461791 6653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:24.479813 6653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:24.481877 6653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:24.482031 6653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:24.485826 6653 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:43465
I20260619 02:53:24.485868 6720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:43465 every 8 connection(s)
I20260619 02:53:24.486191 6653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:24.487003 6721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:24.489270 6721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653: Bootstrap starting.
I20260619 02:53:24.489805 6721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:24.490067 6721 log.cc:824] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:24.490338 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 6653
I20260619 02:53:24.490453 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:24.490631 22163 external_mini_cluster.cc:1428] Setting key 91bc5b13be1d1dc876e915d836001832
I20260619 02:53:24.490846 6721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653: No bootstrap required, opened a new log
I20260619 02:53:24.492623 6721 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec24d75f0c514de88d58e10088a1d653" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43465 } }
I20260619 02:53:24.492815 6721 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:24.492851 6721 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec24d75f0c514de88d58e10088a1d653, State: Initialized, Role: FOLLOWER
I20260619 02:53:24.492972 6721 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec24d75f0c514de88d58e10088a1d653" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43465 } }
I20260619 02:53:24.493042 6721 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:24.493073 6721 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:24.493119 6721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:24.493628 6721 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec24d75f0c514de88d58e10088a1d653" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43465 } }
I20260619 02:53:24.493728 6721 leader_election.cc:302] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec24d75f0c514de88d58e10088a1d653; no voters:
I20260619 02:53:24.493872 6721 leader_election.cc:288] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:24.493906 6726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:24.494014 6726 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [term 1 LEADER]: Becoming Leader. State: Replica: ec24d75f0c514de88d58e10088a1d653, State: Running, Role: LEADER
I20260619 02:53:24.494114 6721 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:24.494158 6726 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec24d75f0c514de88d58e10088a1d653" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43465 } }
I20260619 02:53:24.494513 6726 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec24d75f0c514de88d58e10088a1d653. Latest consensus state: current_term: 1 leader_uuid: "ec24d75f0c514de88d58e10088a1d653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec24d75f0c514de88d58e10088a1d653" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43465 } } }
I20260619 02:53:24.494522 6728 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec24d75f0c514de88d58e10088a1d653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec24d75f0c514de88d58e10088a1d653" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 43465 } } }
I20260619 02:53:24.494693 6728 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:24.494875 6726 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:24.495028 6735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:24.495553 6735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:24.497344 6735 catalog_manager.cc:1347] Generated new cluster ID: 45a0c07f2b8b4c1d9709597a7ff9edd3
I20260619 02:53:24.497424 6735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:53:24 dist-test-slave-jt0t krb5kdc[6640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:24.501181 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:24.491078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34388 (local address 127.21.164.254:43465)
0619 02:53:24.491282 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:24.491290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:24.491317 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:53:24.492003 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:24.492010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:24.492058 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:24.492181 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:24.492769 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.494000 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.495046 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.495647 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.496157 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:24.498270 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:24.498273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:24.498274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:24.498286 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:24.499791 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:24.500100 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:24.500103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:24.500103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:24.500184 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:24.500353 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:24.500353 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:24.500354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:24.500574 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:24.500746 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:24.500816 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 02:53:24.501030 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
I20260619 02:53:24.530395 6735 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:24.531176 6735 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:24.540750 6735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653: Generated new TSK 0
I20260619 02:53:24.540969 6735 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:53:24.565718 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:43465
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:24.646274 6747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:24.646494 6747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:24.646525 6747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:24.647722 6747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:24.647769 6747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:24.647806 6747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:53:24.649363 6747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:43465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:24.649613 6747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:24.649832 6747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:24.654749 6753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:24.654810 6747 server_base.cc:1034] running on GCE node
W20260619 02:53:24.654749 6755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:24.654771 6752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:24.655195 6747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:24.655433 6747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:24.656590 6747 hybrid_clock.cc:648] HybridClock initialized: now 1781837604656577 us; error 26 us; skew 500 ppm
Jun 19 02:53:24 dist-test-slave-jt0t krb5kdc[6640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837604, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:24.659541 6747 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:53:24.659986 6747 webserver.cc:460] Webserver started at http://127.21.164.193:41289/ using document root <none> and password file <none>
I20260619 02:53:24.660188 6747 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:24.660228 6747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:24.660341 6747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:24.661455 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "e1f5ab63e1204708a9e10b0d190adf95"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "265cbae5578fddda03c4b2f0f3725a65"
server_key_iv: "8b912df338c6aac66aa6489c956613fa"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.661792 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "e1f5ab63e1204708a9e10b0d190adf95"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "265cbae5578fddda03c4b2f0f3725a65"
server_key_iv: "8b912df338c6aac66aa6489c956613fa"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.663139 6747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:24.663827 6762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:24.663967 6747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:24.664045 6747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "e1f5ab63e1204708a9e10b0d190adf95"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "265cbae5578fddda03c4b2f0f3725a65"
server_key_iv: "8b912df338c6aac66aa6489c956613fa"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.664119 6747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:24.677270 6747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:24.679244 6747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:24.679369 6747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:24.679590 6747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:24.679901 6747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:24.679934 6747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:24.680011 6747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:24.680039 6747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:24.685729 6747 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:44957
I20260619 02:53:24.685778 6875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:44957 every 8 connection(s)
I20260619 02:53:24.686089 6747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:24.690932 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 6747
I20260619 02:53:24.691051 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:24.691298 22163 external_mini_cluster.cc:1428] Setting key 0c7690cf7da5f7f029ee98dad958704f
Jun 19 02:53:24 dist-test-slave-jt0t krb5kdc[6640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837604, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:24.695776 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:24.686765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:41437 (local address 127.21.164.254:43465)
0619 02:53:24.686821 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:24.686824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:24.687008 (+ 184us) server_negotiation.cc:407] Connection header received
0619 02:53:24.687506 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:24.687508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:24.687520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:24.687567 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:24.688509 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.689091 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.690079 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.690293 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.690348 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:24.692895 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:24.692896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:24.692896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:24.692903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:24.694430 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:24.694883 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:24.694883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:24.694884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:24.694944 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:24.695160 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:24.695160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:24.695161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:24.695283 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:24.695346 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:24.695583 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 02:53:24.695674 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:53:24.695909 6878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:24.686747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:43465 (local address 127.21.164.193:41437)
0619 02:53:24.686890 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:24.686928 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:24.687412 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:24.687649 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:24.687653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:24.688216 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:24.688411 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:24.688416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.689178 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.689179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:24.689997 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:24.689999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.690360 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.690361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:24.690487 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:24.691072 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:24.691083 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:24.692792 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:24.694508 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:24.694514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:24.694516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:24.694819 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:24.695002 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:24.695002 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:24.695003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:24.695108 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:24.695338 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:24.695341 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:24.695464 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 02:53:24.695541 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:53:24.695715 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260619 02:53:24.696326 6876 heartbeater.cc:347] Connected to a master server at 127.21.164.254:43465
I20260619 02:53:24.696434 6876 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:24.696630 6876 heartbeater.cc:511] Master 127.21.164.254:43465 requested a full tablet report, sending...
I20260619 02:53:24.697196 6685 ts_manager.cc:194] Registered new tserver with Master: e1f5ab63e1204708a9e10b0d190adf95 (127.21.164.193:44957)
I20260619 02:53:24.697808 6685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:41437
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:53:24.715894 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:43465
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:24.795251 6881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:24.795483 6881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:24.795508 6881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:24.796717 6881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:24.796761 6881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:24.796788 6881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:53:24.798244 6881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:43465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:24.798462 6881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:24.798676 6881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:24.803401 6886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:24.803444 6889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:24.803397 6887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:24.803620 6881 server_base.cc:1034] running on GCE node
I20260619 02:53:24.803835 6881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:24.804080 6881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:24.805248 6881 hybrid_clock.cc:648] HybridClock initialized: now 1781837604805231 us; error 27 us; skew 500 ppm
Jun 19 02:53:24 dist-test-slave-jt0t krb5kdc[6640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837604, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:24.807689 6881 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:53:24.808135 6881 webserver.cc:460] Webserver started at http://127.21.164.194:38119/ using document root <none> and password file <none>
I20260619 02:53:24.808331 6881 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:24.808377 6881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:24.808501 6881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:24.809562 6881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "412e8615f67c971d511c44fb88b0ae93"
server_key_iv: "21805cc6b0126ac3ffad2badb36f94ad"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.809882 6881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "412e8615f67c971d511c44fb88b0ae93"
server_key_iv: "21805cc6b0126ac3ffad2badb36f94ad"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.811224 6881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:24.812000 6896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:24.812150 6881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:24.812230 6881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "412e8615f67c971d511c44fb88b0ae93"
server_key_iv: "21805cc6b0126ac3ffad2badb36f94ad"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.812296 6881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:24.824222 6881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:24.826437 6881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:24.826594 6881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:24.826826 6881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:24.827132 6881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:24.827167 6881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:24.827198 6881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:24.827219 6881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:24.832571 6881 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:39563
I20260619 02:53:24.832639 7009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:39563 every 8 connection(s)
I20260619 02:53:24.833035 6881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:53:24 dist-test-slave-jt0t krb5kdc[6640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837604, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:24.840641 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 6881
I20260619 02:53:24.840780 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:24.840986 22163 external_mini_cluster.cc:1428] Setting key 6b04ac3fdc56bd377b366ed1a29a84b9
I20260619 02:53:24.842576 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:24.833753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:33027 (local address 127.21.164.254:43465)
0619 02:53:24.833827 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:24.833829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:24.833997 (+ 168us) server_negotiation.cc:407] Connection header received
0619 02:53:24.834552 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:24.834555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:24.834567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:24.834628 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:24.835618 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.836164 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.837319 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.837518 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.837586 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:24.839738 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:24.839740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:24.839740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:24.839748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:24.841265 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:24.841649 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:24.841650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:24.841650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:24.841714 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:24.841977 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:24.841977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:24.841977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:24.842081 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:24.842127 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:24.842429 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 02:53:24.842497 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:53:24.842816 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:24.833726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:43465 (local address 127.21.164.194:33027)
0619 02:53:24.833882 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:24.833919 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:24.834425 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:24.834730 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:24.834735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:24.835326 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:24.835519 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:24.835524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.836261 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.836262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:24.837234 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:24.837237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.837603 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.837604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:24.837715 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:24.838233 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:24.838241 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:24.839647 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:24.841339 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:24.841346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:24.841349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:24.841579 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:24.841773 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:24.841774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:24.841774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:24.841912 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:24.842130 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:24.842135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:24.842302 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 02:53:24.842404 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:53:24.842582 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260619 02:53:24.843309 7010 heartbeater.cc:347] Connected to a master server at 127.21.164.254:43465
I20260619 02:53:24.843405 7010 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:24.843582 7010 heartbeater.cc:511] Master 127.21.164.254:43465 requested a full tablet report, sending...
I20260619 02:53:24.843943 6685 ts_manager.cc:194] Registered new tserver with Master: bf54a76fe9c74f1a93a15af0a29d3f0d (127.21.164.194:39563)
I20260619 02:53:24.844301 6685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:33027
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:53:24.865170 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:43465
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:24.944527 7015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:24.944764 7015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:24.944796 7015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:24.945940 7015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:24.945979 7015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:24.946013 7015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:53:24.947436 7015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:43465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:24.947672 7015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:24.947878 7015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:24.952557 7020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:24.952574 7023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:24.952557 7021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:24.952581 7015 server_base.cc:1034] running on GCE node
I20260619 02:53:24.953022 7015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:24.953262 7015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:24.954397 7015 hybrid_clock.cc:648] HybridClock initialized: now 1781837604954384 us; error 28 us; skew 500 ppm
Jun 19 02:53:24 dist-test-slave-jt0t krb5kdc[6640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837604, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:24.956775 7015 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:53:24.957161 7015 webserver.cc:460] Webserver started at http://127.21.164.195:41659/ using document root <none> and password file <none>
I20260619 02:53:24.957345 7015 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:24.957382 7015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:24.957474 7015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:24.958462 7015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "257d490f1b9c4ac5a8b5f73bd051c75e"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "e6f8da1e1a29f7cd390827be08dfa5d1"
server_key_iv: "5b43cd61fce7c57fc990f7b1e61d03e4"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.958758 7015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "257d490f1b9c4ac5a8b5f73bd051c75e"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "e6f8da1e1a29f7cd390827be08dfa5d1"
server_key_iv: "5b43cd61fce7c57fc990f7b1e61d03e4"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.959975 7015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:24.960621 7030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:24.960776 7015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:24.960856 7015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "257d490f1b9c4ac5a8b5f73bd051c75e"
format_stamp: "Formatted at 2026-06-19 02:53:24 on dist-test-slave-jt0t"
server_key: "e6f8da1e1a29f7cd390827be08dfa5d1"
server_key_iv: "5b43cd61fce7c57fc990f7b1e61d03e4"
server_key_version: "encryptionkey@0"
I20260619 02:53:24.960916 7015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:24.984839 7015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:24.986807 7015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:24.986944 7015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:24.987164 7015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:24.987457 7015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:24.987488 7015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:24.987517 7015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:24.987540 7015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:24.993005 7015 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:41719
I20260619 02:53:24.993054 7143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:41719 every 8 connection(s)
I20260619 02:53:24.993356 7015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:53:24 dist-test-slave-jt0t krb5kdc[6640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837604, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:25.000118 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 7015
I20260619 02:53:25.000223 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:25.000437 22163 external_mini_cluster.cc:1428] Setting key ccd2f0343003dde713220d9422f58ffb
I20260619 02:53:25.002612 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:24.994038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:34923 (local address 127.21.164.254:43465)
0619 02:53:24.994099 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:24.994101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:24.994290 (+ 189us) server_negotiation.cc:407] Connection header received
0619 02:53:24.994919 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:24.994920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:24.994933 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:24.994982 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:24.995966 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.996469 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.997476 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.997687 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.997744 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:24.999796 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:24.999798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:24.999798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:24.999807 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:25.001397 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:25.001747 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:25.001748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:25.001748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:25.001805 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:25.002018 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:25.002019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:25.002020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:25.002133 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:25.002186 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.002441 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.002516 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:53:25.002779 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:24.994009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:43465 (local address 127.21.164.195:34923)
0619 02:53:24.994178 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:24.994213 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:24.994812 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:24.995052 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:24.995056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:24.995649 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:24.995873 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:24.995878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.996548 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.996549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:24.997397 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:24.997400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:24.997755 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:24.997756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:24.997891 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:24.998329 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:24.998337 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:24.999703 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:25.001454 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:25.001461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:25.001463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:25.001681 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:25.001857 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:25.001858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:25.001858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:25.001964 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:25.002186 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:25.002192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:25.002326 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.002414 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.002572 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260619 02:53:25.003150 7144 heartbeater.cc:347] Connected to a master server at 127.21.164.254:43465
I20260619 02:53:25.003242 7144 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:25.003402 7144 heartbeater.cc:511] Master 127.21.164.254:43465 requested a full tablet report, sending...
I20260619 02:53:25.003717 6685 ts_manager.cc:194] Registered new tserver with Master: 257d490f1b9c4ac5a8b5f73bd051c75e (127.21.164.195:41719)
I20260619 02:53:25.004083 6685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:34923
I20260619 02:53:25.012286 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:25.019171 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.013576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.21.164.254:43465)
0619 02:53:25.013643 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.013645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.013728 (+ 83us) server_negotiation.cc:407] Connection header received
0619 02:53:25.013768 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.013769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.013778 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.013832 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:25.014320 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.014812 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.015590 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.015778 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.015835 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.016960 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:25.016961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:25.016961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:25.016967 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:25.018231 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:25.018528 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:25.018528 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:25.018529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:25.018572 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:25.018700 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:25.018701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:25.018701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:25.018831 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:25.018901 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.019045 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.019104 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:53:25.020728 6685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:25.022228 6685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:25.028234 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.024245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:44957 (local address 127.0.0.1:50054)
0619 02:53:25.024404 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.024442 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.024504 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.024869 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.024872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.024874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:25.024972 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.024974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.026246 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.026247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.027304 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.027306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.027938 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.027939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.028020 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.028032 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.028096 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.028153 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":51,"thread_start_us":58,"threads_started":1}
I20260619 02:53:25.028242 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.024418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.21.164.193:44957)
0619 02:53:25.024715 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.024720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.024729 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:25.024765 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.024767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.024782 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.024849 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:25.025064 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.026146 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.027372 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.027864 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.027930 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.027988 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.028116 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.028154 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":80,"threads_started":1}
I20260619 02:53:25.028532 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.024502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.21.164.195:41719)
0619 02:53:25.024730 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.024735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.024743 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:25.024778 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.024780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.024791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.024857 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:25.025040 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.025848 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.027232 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.027950 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.028020 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.028070 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.028403 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.028446 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260619 02:53:25.028333 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.024246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:41719 (local address 127.0.0.1:47234)
0619 02:53:25.024599 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.024609 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.024685 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.024869 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.024871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.024873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:25.024958 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.024960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.025932 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.025933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.027153 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.027156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.028103 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.028104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.028165 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.028169 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.028211 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.028256 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":8,"thread_start_us":60,"threads_started":1}
I20260619 02:53:25.028611 7156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.024509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.21.164.194:39563)
0619 02:53:25.024989 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.024993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.025001 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:25.025037 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.025039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.025081 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.025149 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:25.025343 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.026351 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.027442 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.028129 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.028185 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.028230 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.028498 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.028538 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":67,"threads_started":1}
I20260619 02:53:25.028446 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.024246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:39563 (local address 127.0.0.1:60432)
0619 02:53:25.024601 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.024609 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.024642 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.025224 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.025225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.025227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:25.025262 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.025264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.026433 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.026434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.027357 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.027359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.028259 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.028259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.028312 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.028316 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.028356 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.028392 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":24,"thread_start_us":50,"threads_started":1}
I20260619 02:53:25.029455 6944 tablet_service.cc:1467] Processing CreateTablet for tablet 483bb3e1548f4dc885dd2f2460666efb (DEFAULT_TABLE table=test-table [id=c6382845384b4594af84ede5ed196ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:25.029465 7078 tablet_service.cc:1467] Processing CreateTablet for tablet 483bb3e1548f4dc885dd2f2460666efb (DEFAULT_TABLE table=test-table [id=c6382845384b4594af84ede5ed196ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:25.029461 6810 tablet_service.cc:1467] Processing CreateTablet for tablet 483bb3e1548f4dc885dd2f2460666efb (DEFAULT_TABLE table=test-table [id=c6382845384b4594af84ede5ed196ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:25.029795 6944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483bb3e1548f4dc885dd2f2460666efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:25.029798 6810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483bb3e1548f4dc885dd2f2460666efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:25.029798 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483bb3e1548f4dc885dd2f2460666efb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:25.032253 7158 tablet_bootstrap.cc:492] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: Bootstrap starting.
I20260619 02:53:25.032953 7158 tablet_bootstrap.cc:654] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:25.033121 7159 tablet_bootstrap.cc:492] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95: Bootstrap starting.
I20260619 02:53:25.033452 7158 log.cc:824] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:25.033617 7159 tablet_bootstrap.cc:654] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:25.033766 7160 tablet_bootstrap.cc:492] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e: Bootstrap starting.
I20260619 02:53:25.034190 7158 tablet_bootstrap.cc:492] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: No bootstrap required, opened a new log
I20260619 02:53:25.034197 7159 log.cc:824] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:25.034261 7158 ts_tablet_manager.cc:1397] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:25.034272 7160 tablet_bootstrap.cc:654] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:25.034770 7160 log.cc:824] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:25.034938 7159 tablet_bootstrap.cc:492] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95: No bootstrap required, opened a new log
I20260619 02:53:25.035005 7159 ts_tablet_manager.cc:1397] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:25.035362 7160 tablet_bootstrap.cc:492] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e: No bootstrap required, opened a new log
I20260619 02:53:25.035419 7160 ts_tablet_manager.cc:1397] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:25.036129 7158 raft_consensus.cc:348] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1f5ab63e1204708a9e10b0d190adf95" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44957 } } peers { permanent_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39563 } } peers { permanent_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41719 } }
I20260619 02:53:25.036314 7158 raft_consensus.cc:374] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:25.036357 7158 raft_consensus.cc:729] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf54a76fe9c74f1a93a15af0a29d3f0d, State: Initialized, Role: FOLLOWER
I20260619 02:53:25.036516 7158 consensus_queue.cc:260] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1f5ab63e1204708a9e10b0d190adf95" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44957 } } peers { permanent_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39563 } } peers { permanent_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41719 } }
I20260619 02:53:25.036820 7158 ts_tablet_manager.cc:1428] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:25.036958 7010 heartbeater.cc:503] Master 127.21.164.254:43465 was elected leader, sending a full tablet report...
I20260619 02:53:25.036870 7159 raft_consensus.cc:348] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1f5ab63e1204708a9e10b0d190adf95" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44957 } } peers { permanent_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39563 } } peers { permanent_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41719 } }
I20260619 02:53:25.037029 7159 raft_consensus.cc:374] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:25.037051 7159 raft_consensus.cc:729] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f5ab63e1204708a9e10b0d190adf95, State: Initialized, Role: FOLLOWER
I20260619 02:53:25.037132 7159 consensus_queue.cc:260] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1f5ab63e1204708a9e10b0d190adf95" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44957 } } peers { permanent_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39563 } } peers { permanent_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41719 } }
I20260619 02:53:25.037120 7160 raft_consensus.cc:348] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1f5ab63e1204708a9e10b0d190adf95" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44957 } } peers { permanent_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39563 } } peers { permanent_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41719 } }
I20260619 02:53:25.037271 7160 raft_consensus.cc:374] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:25.037305 7160 raft_consensus.cc:729] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257d490f1b9c4ac5a8b5f73bd051c75e, State: Initialized, Role: FOLLOWER
I20260619 02:53:25.037369 7159 ts_tablet_manager.cc:1428] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 02:53:25.037407 7160 consensus_queue.cc:260] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1f5ab63e1204708a9e10b0d190adf95" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44957 } } peers { permanent_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39563 } } peers { permanent_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41719 } }
I20260619 02:53:25.037544 6876 heartbeater.cc:503] Master 127.21.164.254:43465 was elected leader, sending a full tablet report...
I20260619 02:53:25.037648 7144 heartbeater.cc:503] Master 127.21.164.254:43465 was elected leader, sending a full tablet report...
I20260619 02:53:25.037649 7160 ts_tablet_manager.cc:1428] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260619 02:53:25.083604 7011 tablet.cc:2369] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:25.117710 7165 raft_consensus.cc:482] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:25.117837 7165 raft_consensus.cc:504] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1f5ab63e1204708a9e10b0d190adf95" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44957 } } peers { permanent_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39563 } } peers { permanent_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41719 } }
I20260619 02:53:25.118232 7165 leader_election.cc:288] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf54a76fe9c74f1a93a15af0a29d3f0d (127.21.164.194:39563), 257d490f1b9c4ac5a8b5f73bd051c75e (127.21.164.195:41719)
I20260619 02:53:25.122237 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.118501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:41455 (local address 127.21.164.195:41719)
0619 02:53:25.118559 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.118561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.118797 (+ 236us) server_negotiation.cc:407] Connection header received
0619 02:53:25.118848 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.118849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.118857 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.118903 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:25.119044 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.119741 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.121208 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.121819 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.121899 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.121908 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.122127 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.122169 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:53:25.122237 7156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.118558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:52217 (local address 127.21.164.194:39563)
0619 02:53:25.118604 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.118606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.118616 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:53:25.118648 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.118649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.118661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.118736 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:25.118901 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.119703 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.121208 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.121819 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.121900 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.121908 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.122115 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.122166 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:53:25.122238 6878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.118455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:39563 (local address 127.21.164.193:52217)
0619 02:53:25.118506 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.118516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.118560 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.118752 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.118753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.118755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:25.118830 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.118833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.119796 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.119798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.121107 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.121110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.121916 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.121918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.122007 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.122010 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.122070 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.122120 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":43}
I20260619 02:53:25.122247 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.118455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:41719 (local address 127.21.164.193:41455)
0619 02:53:25.118734 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.118742 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.118802 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.118915 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.118916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.118917 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:25.118967 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.118968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.119824 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.119825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.121107 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.121110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.121916 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.121918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.122005 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.122010 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.122091 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.122122 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":110,"thread_start_us":129,"threads_started":1}
I20260619 02:53:25.122684 6964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483bb3e1548f4dc885dd2f2460666efb" candidate_uuid: "e1f5ab63e1204708a9e10b0d190adf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" is_pre_election: true
I20260619 02:53:25.122685 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483bb3e1548f4dc885dd2f2460666efb" candidate_uuid: "e1f5ab63e1204708a9e10b0d190adf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" is_pre_election: true
I20260619 02:53:25.122823 7098 raft_consensus.cc:2454] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1f5ab63e1204708a9e10b0d190adf95 in term 0.
I20260619 02:53:25.122826 6964 raft_consensus.cc:2454] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1f5ab63e1204708a9e10b0d190adf95 in term 0.
I20260619 02:53:25.123052 6763 leader_election.cc:302] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf54a76fe9c74f1a93a15af0a29d3f0d, e1f5ab63e1204708a9e10b0d190adf95; no voters:
I20260619 02:53:25.123230 7165 raft_consensus.cc:2781] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:25.123291 7165 raft_consensus.cc:482] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:25.123312 7165 raft_consensus.cc:3037] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:25.123996 7165 raft_consensus.cc:504] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1f5ab63e1204708a9e10b0d190adf95" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44957 } } peers { permanent_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39563 } } peers { permanent_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41719 } }
I20260619 02:53:25.124166 7165 leader_election.cc:288] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [CANDIDATE]: Term 1 election: Requested vote from peers bf54a76fe9c74f1a93a15af0a29d3f0d (127.21.164.194:39563), 257d490f1b9c4ac5a8b5f73bd051c75e (127.21.164.195:41719)
I20260619 02:53:25.124325 6964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483bb3e1548f4dc885dd2f2460666efb" candidate_uuid: "e1f5ab63e1204708a9e10b0d190adf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d"
I20260619 02:53:25.124352 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483bb3e1548f4dc885dd2f2460666efb" candidate_uuid: "e1f5ab63e1204708a9e10b0d190adf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e"
I20260619 02:53:25.124411 6964 raft_consensus.cc:3037] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:25.124418 7098 raft_consensus.cc:3037] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:25.125101 7098 raft_consensus.cc:2454] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1f5ab63e1204708a9e10b0d190adf95 in term 1.
I20260619 02:53:25.125223 6964 raft_consensus.cc:2454] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1f5ab63e1204708a9e10b0d190adf95 in term 1.
I20260619 02:53:25.125291 6766 leader_election.cc:302] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 257d490f1b9c4ac5a8b5f73bd051c75e, e1f5ab63e1204708a9e10b0d190adf95; no voters:
I20260619 02:53:25.125391 7165 raft_consensus.cc:2781] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:25.125542 7165 raft_consensus.cc:686] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [term 1 LEADER]: Becoming Leader. State: Replica: e1f5ab63e1204708a9e10b0d190adf95, State: Running, Role: LEADER
I20260619 02:53:25.125634 7165 consensus_queue.cc:237] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1f5ab63e1204708a9e10b0d190adf95" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44957 } } peers { permanent_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39563 } } peers { permanent_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41719 } }
I20260619 02:53:25.126439 6683 catalog_manager.cc:5495] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1f5ab63e1204708a9e10b0d190adf95 (127.21.164.193). New cstate: current_term: 1 leader_uuid: "e1f5ab63e1204708a9e10b0d190adf95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1f5ab63e1204708a9e10b0d190adf95" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41719 } health_report { overall_health: UNKNOWN } } }
I20260619 02:53:25.147473 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.145048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.21.164.193:44957)
0619 02:53:25.145106 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.145108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.145116 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:25.145228 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.145230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.145258 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.145347 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:25.145502 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.146019 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.146767 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.146952 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.147013 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.147155 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:25.147263 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:25.147334 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.147353 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.147374 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:53:25.149302 7098 raft_consensus.cc:1261] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e [term 1 FOLLOWER]: Refusing update from remote peer e1f5ab63e1204708a9e10b0d190adf95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:25.149335 6964 raft_consensus.cc:1261] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d [term 1 FOLLOWER]: Refusing update from remote peer e1f5ab63e1204708a9e10b0d190adf95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:25.149629 7168 consensus_queue.cc:1035] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf54a76fe9c74f1a93a15af0a29d3f0d" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:25.149758 7165 consensus_queue.cc:1035] T 483bb3e1548f4dc885dd2f2460666efb P e1f5ab63e1204708a9e10b0d190adf95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "257d490f1b9c4ac5a8b5f73bd051c75e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:25.152096 7171 mvcc.cc:205] Tried to move back new op lower bound from 7298406830688874496 to 7298406830595235840. Current Snapshot: MvccSnapshot[applied={T|T < 7298406830688874496}]
I20260619 02:53:25.152436 7173 mvcc.cc:205] Tried to move back new op lower bound from 7298406830688874496 to 7298406830595235840. Current Snapshot: MvccSnapshot[applied={T|T < 7298406830688874496}]
I20260619 02:53:25.152444 7172 mvcc.cc:205] Tried to move back new op lower bound from 7298406830688874496 to 7298406830595235840. Current Snapshot: MvccSnapshot[applied={T|T < 7298406830688874496}]
I20260619 02:53:25.154597 6684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:25.154698 6684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:25.155431 6684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec24d75f0c514de88d58e10088a1d653: Sending DeleteTablet for 3 replicas of tablet 483bb3e1548f4dc885dd2f2460666efb
I20260619 02:53:25.155761 6810 tablet_service.cc:1514] Processing DeleteTablet for tablet 483bb3e1548f4dc885dd2f2460666efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:25 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:50054
I20260619 02:53:25.155763 7078 tablet_service.cc:1514] Processing DeleteTablet for tablet 483bb3e1548f4dc885dd2f2460666efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:25 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:47234
I20260619 02:53:25.155995 7180 tablet_replica.cc:330] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e: stopping tablet replica
I20260619 02:53:25.155983 6944 tablet_service.cc:1514] Processing DeleteTablet for tablet 483bb3e1548f4dc885dd2f2460666efb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:25 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:60432
I20260619 02:53:25.156185 7180 raft_consensus.cc:2229] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:25.156294 7182 tablet_replica.cc:330] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: stopping tablet replica
I20260619 02:53:25.156315 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 6747
I20260619 02:53:25.156361 7180 raft_consensus.cc:2258] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:25.157644 7182 raft_consensus.cc:2229] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:25.157729 7180 ts_tablet_manager.cc:1905] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:25.157833 7182 raft_consensus.cc:2258] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:25.158267 7182 ts_tablet_manager.cc:1905] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:25.159521 7182 ts_tablet_manager.cc:1918] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:25.159518 7180 ts_tablet_manager.cc:1918] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:25.159583 7180 log.cc:1196] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/483bb3e1548f4dc885dd2f2460666efb
I20260619 02:53:25.159591 7182 log.cc:1196] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/483bb3e1548f4dc885dd2f2460666efb
I20260619 02:53:25.159828 7180 ts_tablet_manager.cc:1939] T 483bb3e1548f4dc885dd2f2460666efb P 257d490f1b9c4ac5a8b5f73bd051c75e: Deleting consensus metadata
I20260619 02:53:25.159837 7182 ts_tablet_manager.cc:1939] T 483bb3e1548f4dc885dd2f2460666efb P bf54a76fe9c74f1a93a15af0a29d3f0d: Deleting consensus metadata
I20260619 02:53:25.160228 6669 catalog_manager.cc:4841] TS bf54a76fe9c74f1a93a15af0a29d3f0d (127.21.164.194:39563): tablet 483bb3e1548f4dc885dd2f2460666efb (table test-table [id=c6382845384b4594af84ede5ed196ba5]) successfully deleted
I20260619 02:53:25.160275 6672 catalog_manager.cc:4841] TS 257d490f1b9c4ac5a8b5f73bd051c75e (127.21.164.195:41719): tablet 483bb3e1548f4dc885dd2f2460666efb (table test-table [id=c6382845384b4594af84ede5ed196ba5]) successfully deleted
W20260619 02:53:25.161515 6671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:44957 (error 108)
I20260619 02:53:25.161659 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 6881
I20260619 02:53:25.162453 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.162320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:44957 (local address 127.0.0.1:50076)
0619 02:53:25.162374 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.162420 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:44957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 02:53:25.162598 6671 catalog_manager.cc:4584] TS e1f5ab63e1204708a9e10b0d190adf95 (127.21.164.193:44957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 483bb3e1548f4dc885dd2f2460666efb: Network error: Client connection negotiation failed: client connection to 127.21.164.193:44957: connect: Connection refused (error 111)
I20260619 02:53:25.166581 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 7015
I20260619 02:53:25.170593 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 6653
2026-06-19T02:53:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1879 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T02:53:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:25Z Disabled control of system clock
I20260619 02:53:25.195575 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:36521
--webserver_interface=127.21.164.254
--webserver_port=0
--builtin_ntp_servers=127.21.164.212:46259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:25.273989 7187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:25.274201 7187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:25.274221 7187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:25.275329 7187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:25.275364 7187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:25.275375 7187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:25.275386 7187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:25.275396 7187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:25.276701 7187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:46259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:36521
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:25.276895 7187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:25.277127 7187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:25.281817 7195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:25.281821 7192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:25.281816 7193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:25.282020 7187 server_base.cc:1034] running on GCE node
I20260619 02:53:25.282334 7187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:25.282618 7187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:25.283777 7187 hybrid_clock.cc:648] HybridClock initialized: now 1781837605283759 us; error 32 us; skew 500 ppm
I20260619 02:53:25.285032 7187 webserver.cc:460] Webserver started at http://127.21.164.254:42673/ using document root <none> and password file <none>
I20260619 02:53:25.285279 7187 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:25.285329 7187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:25.285454 7187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:25.286525 7187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "c767abb0e2ec42678b446effee572bcf"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "4062a8cc5c67defb18f51c474066a4bc"
server_key_iv: "ab154499c7ba9ff9dfdbcece46880327"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.286854 7187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "c767abb0e2ec42678b446effee572bcf"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "4062a8cc5c67defb18f51c474066a4bc"
server_key_iv: "ab154499c7ba9ff9dfdbcece46880327"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.288139 7187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:25.288894 7201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:25.289117 7187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:25.289204 7187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "c767abb0e2ec42678b446effee572bcf"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "4062a8cc5c67defb18f51c474066a4bc"
server_key_iv: "ab154499c7ba9ff9dfdbcece46880327"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.289295 7187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:25.298121 7187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:25.298401 7187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:25.298524 7187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:25.302131 7187 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:36521
I20260619 02:53:25.302181 7253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:36521 every 8 connection(s)
I20260619 02:53:25.302477 7187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:25.303397 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:25.305648 7254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf: Bootstrap starting.
I20260619 02:53:25.306186 7254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:25.306439 7254 log.cc:824] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:25.307055 7254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf: No bootstrap required, opened a new log
I20260619 02:53:25.308560 7254 raft_consensus.cc:348] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c767abb0e2ec42678b446effee572bcf" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36521 } }
I20260619 02:53:25.308738 7254 raft_consensus.cc:374] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:25.308765 7254 raft_consensus.cc:729] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c767abb0e2ec42678b446effee572bcf, State: Initialized, Role: FOLLOWER
I20260619 02:53:25.308910 7254 consensus_queue.cc:260] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c767abb0e2ec42678b446effee572bcf" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36521 } }
I20260619 02:53:25.308996 7254 raft_consensus.cc:388] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:25.309024 7254 raft_consensus.cc:482] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:25.309067 7254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:25.309578 7254 raft_consensus.cc:504] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c767abb0e2ec42678b446effee572bcf" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36521 } }
I20260619 02:53:25.309645 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 7187
I20260619 02:53:25.309674 7254 leader_election.cc:302] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c767abb0e2ec42678b446effee572bcf; no voters:
I20260619 02:53:25.309733 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:25.309834 7254 leader_election.cc:288] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:25.309882 7257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:25.309928 22163 external_mini_cluster.cc:1428] Setting key 6a4882e6764df4d132df366d6a4c8e96
I20260619 02:53:25.310058 7257 raft_consensus.cc:686] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [term 1 LEADER]: Becoming Leader. State: Replica: c767abb0e2ec42678b446effee572bcf, State: Running, Role: LEADER
I20260619 02:53:25.310117 7254 sys_catalog.cc:564] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:25.310148 7257 consensus_queue.cc:237] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c767abb0e2ec42678b446effee572bcf" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36521 } }
I20260619 02:53:25.310612 7258 sys_catalog.cc:455] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c767abb0e2ec42678b446effee572bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c767abb0e2ec42678b446effee572bcf" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36521 } } }
I20260619 02:53:25.310700 7258 sys_catalog.cc:458] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:25.310860 7259 sys_catalog.cc:455] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c767abb0e2ec42678b446effee572bcf. Latest consensus state: current_term: 1 leader_uuid: "c767abb0e2ec42678b446effee572bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c767abb0e2ec42678b446effee572bcf" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 36521 } } }
I20260619 02:53:25.310897 7266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:25.311473 7266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:25.310956 7259 sys_catalog.cc:458] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:25.313092 7266 catalog_manager.cc:1347] Generated new cluster ID: f51733c742ac49c698d419bd4d8ffebd
I20260619 02:53:25.313144 7266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:25.315606 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.310515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.21.164.254:36521)
0619 02:53:25.311210 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.311220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.311254 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:53:25.311422 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.311428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.311500 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.311643 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:25.311837 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.313037 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.313779 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.314190 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.314297 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.314962 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:25.314966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:25.314966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:25.315128 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:25.315160 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:25.315171 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:25.315223 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.315293 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.315445 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":130,"threads_started":1}
I20260619 02:53:25.329421 7266 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:25.329921 7266 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:25.334940 7266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf: Generated new TSK 0
I20260619 02:53:25.335126 7266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:25.337051 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36521
--builtin_ntp_servers=127.21.164.212:46259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:25.455631 7278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:25.455946 7278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:25.455986 7278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:25.457850 7278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:25.457923 7278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:25.457964 7278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:53:25.460016 7278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:46259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:25.460299 7278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:25.460707 7278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:25.466032 7286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:25.466045 7284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:25.466044 7283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:25.466435 7278 server_base.cc:1034] running on GCE node
I20260619 02:53:25.466634 7278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:25.466872 7278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:25.468072 7278 hybrid_clock.cc:648] HybridClock initialized: now 1781837605468057 us; error 31 us; skew 500 ppm
I20260619 02:53:25.469353 7278 webserver.cc:460] Webserver started at http://127.21.164.193:42225/ using document root <none> and password file <none>
I20260619 02:53:25.469604 7278 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:25.469655 7278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:25.469785 7278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:25.470880 7278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "373ed8ea35224ce9a9036e445a0d5d20"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "58676167e4982c794c8a31aab5ba8a49"
server_key_iv: "db8e867f3bfb96f1ffb935511bd8784d"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.471223 7278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "373ed8ea35224ce9a9036e445a0d5d20"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "58676167e4982c794c8a31aab5ba8a49"
server_key_iv: "db8e867f3bfb96f1ffb935511bd8784d"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.472545 7278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:53:25.473414 7292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:25.473567 7278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:53:25.473651 7278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "373ed8ea35224ce9a9036e445a0d5d20"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "58676167e4982c794c8a31aab5ba8a49"
server_key_iv: "db8e867f3bfb96f1ffb935511bd8784d"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.473718 7278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:25.492426 7278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:25.492796 7278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:25.492930 7278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:25.493167 7278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:25.493489 7278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:25.493522 7278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:25.493552 7278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:25.493587 7278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:25.498791 7278 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:36771
I20260619 02:53:25.498857 7405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:36771 every 8 connection(s)
I20260619 02:53:25.499159 7278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:25.502907 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 7278
I20260619 02:53:25.503026 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:25.503237 22163 external_mini_cluster.cc:1428] Setting key 724d4b4dceb2065366a01b809f90a063
I20260619 02:53:25.504230 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36521
--builtin_ntp_servers=127.21.164.212:46259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:53:25.504797 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.499844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:50171 (local address 127.21.164.254:36521)
0619 02:53:25.499904 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.499906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.500192 (+ 286us) server_negotiation.cc:407] Connection header received
0619 02:53:25.500703 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.500705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.500720 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.500775 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:25.501477 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.502000 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.503241 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.503521 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.503590 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.504511 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:25.504512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:25.504513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:25.504517 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:25.504532 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:25.504536 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:25.504572 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.504699 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.504734 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:53:25.505039 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.499838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:36521 (local address 127.21.164.193:50171)
0619 02:53:25.500017 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.500052 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.500580 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.500837 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.500840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.500849 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:25.501366 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.501371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.502114 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.502115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.503145 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.503149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.503622 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.503624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.503784 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.504346 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:25.504347 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:25.504387 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:25.504417 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:25.504587 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:25.504591 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:25.504594 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.504698 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.504823 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260619 02:53:25.505841 7406 heartbeater.cc:347] Connected to a master server at 127.21.164.254:36521
I20260619 02:53:25.505926 7406 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:25.506104 7406 heartbeater.cc:511] Master 127.21.164.254:36521 requested a full tablet report, sending...
I20260619 02:53:25.506672 7218 ts_manager.cc:194] Registered new tserver with Master: 373ed8ea35224ce9a9036e445a0d5d20 (127.21.164.193:36771)
I20260619 02:53:25.507303 7218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.193:50171
W20260619 02:53:25.586275 7409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:25.586499 7409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:25.586530 7409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:25.587719 7409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:25.587760 7409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:25.587795 7409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:53:25.589244 7409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:46259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:25.589493 7409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:25.589787 7409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:25.594910 7414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:25.594928 7409 server_base.cc:1034] running on GCE node
W20260619 02:53:25.594913 7415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:25.595255 7417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:25.595533 7409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:25.595851 7409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:25.597062 7409 hybrid_clock.cc:648] HybridClock initialized: now 1781837605597044 us; error 29 us; skew 500 ppm
I20260619 02:53:25.598354 7409 webserver.cc:460] Webserver started at http://127.21.164.194:38259/ using document root <none> and password file <none>
I20260619 02:53:25.598595 7409 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:25.598640 7409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:25.598748 7409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:25.599826 7409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "8ef305be3218444d86da6f4477102f69"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "7cdd218708cd295b8f7fdfff535d79fd"
server_key_iv: "3628caedf9c70f4823b9ec44dd0d6bbc"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.600148 7409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "8ef305be3218444d86da6f4477102f69"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "7cdd218708cd295b8f7fdfff535d79fd"
server_key_iv: "3628caedf9c70f4823b9ec44dd0d6bbc"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.601418 7409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:53:25.602133 7423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:25.602277 7409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:53:25.602351 7409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "8ef305be3218444d86da6f4477102f69"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "7cdd218708cd295b8f7fdfff535d79fd"
server_key_iv: "3628caedf9c70f4823b9ec44dd0d6bbc"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.602420 7409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:25.630910 7409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:25.631209 7409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:25.631340 7409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:25.631568 7409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:25.631896 7409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:25.631928 7409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:25.631958 7409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:25.631974 7409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:25.637228 7409 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:43455
I20260619 02:53:25.637305 7536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:43455 every 8 connection(s)
I20260619 02:53:25.637604 7409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:25.639492 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 7409
I20260619 02:53:25.639612 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:25.639782 22163 external_mini_cluster.cc:1428] Setting key 56f70bad22e70371a555f5d5797753d7
I20260619 02:53:25.640682 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36521
--builtin_ntp_servers=127.21.164.212:46259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:53:25.642958 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.638301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:46237 (local address 127.21.164.254:36521)
0619 02:53:25.638360 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.638362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.638593 (+ 231us) server_negotiation.cc:407] Connection header received
0619 02:53:25.639145 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.639146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.639157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.639210 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:25.639909 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.640595 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.641640 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.641822 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.641880 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.642701 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:25.642703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:25.642703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:25.642709 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:25.642724 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:25.642727 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:25.642769 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.642845 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.642877 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:53:25.643178 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.638294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:36521 (local address 127.21.164.194:46237)
0619 02:53:25.638476 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.638510 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.639026 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.639276 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.639279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.639288 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:25.639827 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.639833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.640702 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.640703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.641548 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.641551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.641892 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.641893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.642038 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.642558 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:25.642558 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:25.642589 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:25.642614 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:25.642775 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:25.642781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:25.642784 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.642849 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.642954 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260619 02:53:25.643649 7537 heartbeater.cc:347] Connected to a master server at 127.21.164.254:36521
I20260619 02:53:25.643744 7537 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:25.643923 7537 heartbeater.cc:511] Master 127.21.164.254:36521 requested a full tablet report, sending...
I20260619 02:53:25.644280 7218 ts_manager.cc:194] Registered new tserver with Master: 8ef305be3218444d86da6f4477102f69 (127.21.164.194:43455)
I20260619 02:53:25.644675 7218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.194:46237
W20260619 02:53:25.720183 7540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:25.720412 7540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:25.720443 7540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:25.721629 7540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:25.721670 7540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:25.721704 7540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:53:25.723085 7540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:46259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:36521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:25.723325 7540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:25.723603 7540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:25.728425 7545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:25.728444 7548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:25.728446 7546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:25.728461 7540 server_base.cc:1034] running on GCE node
I20260619 02:53:25.728866 7540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:25.729128 7540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:25.730294 7540 hybrid_clock.cc:648] HybridClock initialized: now 1781837605730280 us; error 23 us; skew 500 ppm
I20260619 02:53:25.731549 7540 webserver.cc:460] Webserver started at http://127.21.164.195:40839/ using document root <none> and password file <none>
I20260619 02:53:25.731774 7540 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:25.731822 7540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:25.731945 7540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:25.733054 7540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "0b71ad5f19c941c490a9ad8f0b492ad7"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "6bfe8ae576863101c7d9a844c9629cf8"
server_key_iv: "826b921316865f53cd7534d9482a0011"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.733381 7540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "0b71ad5f19c941c490a9ad8f0b492ad7"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "6bfe8ae576863101c7d9a844c9629cf8"
server_key_iv: "826b921316865f53cd7534d9482a0011"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.734593 7540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:25.735285 7554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:25.735440 7540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:25.735518 7540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "0b71ad5f19c941c490a9ad8f0b492ad7"
format_stamp: "Formatted at 2026-06-19 02:53:25 on dist-test-slave-jt0t"
server_key: "6bfe8ae576863101c7d9a844c9629cf8"
server_key_iv: "826b921316865f53cd7534d9482a0011"
server_key_version: "encryptionkey@0"
I20260619 02:53:25.735581 7540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:25.750985 7540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:25.751303 7540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:25.751434 7540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:25.751644 7540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:25.751976 7540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:25.752015 7540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:25.752045 7540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:25.752068 7540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:25.757493 7540 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:45845
I20260619 02:53:25.757541 7667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:45845 every 8 connection(s)
I20260619 02:53:25.757884 7540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:53:25.762897 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.758913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:53399 (local address 127.21.164.254:36521)
0619 02:53:25.758983 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.758985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.759194 (+ 209us) server_negotiation.cc:407] Connection header received
0619 02:53:25.759664 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.759665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.759676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.759731 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:25.760341 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.760887 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.761781 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.761956 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.761999 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.762695 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:25.762697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:25.762697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:25.762701 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:25.762717 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:25.762721 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:25.762758 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.762811 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.762837 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:53:25.763082 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.758899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:36521 (local address 127.21.164.195:53399)
0619 02:53:25.759081 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.759114 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.759588 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.759785 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.759788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.759797 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:25.760281 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.760285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.760949 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.760949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.761720 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.761722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.762007 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.762008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.762117 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.762576 (+ 459us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:25.762576 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:25.762604 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:25.762626 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:25.762762 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:25.762765 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:25.762768 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.762811 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.762886 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260619 02:53:25.763478 7668 heartbeater.cc:347] Connected to a master server at 127.21.164.254:36521
I20260619 02:53:25.763572 7668 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:25.763737 7668 heartbeater.cc:511] Master 127.21.164.254:36521 requested a full tablet report, sending...
I20260619 02:53:25.764060 7218 ts_manager.cc:194] Registered new tserver with Master: 0b71ad5f19c941c490a9ad8f0b492ad7 (127.21.164.195:45845)
I20260619 02:53:25.764425 7218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.195:53399
I20260619 02:53:25.765336 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 7540
I20260619 02:53:25.765416 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:25.765594 22163 external_mini_cluster.cc:1428] Setting key 41d4a0cf5cac1b2bedf3826ee348b6d2
I20260619 02:53:25.766271 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:25.770117 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.767493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.21.164.254:36521)
0619 02:53:25.767567 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.767568 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.767702 (+ 134us) server_negotiation.cc:407] Connection header received
0619 02:53:25.767760 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.767761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.767771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.767802 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:25.767922 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.768385 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.769166 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.769352 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.769411 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.769912 (+ 501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:25.769913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:25.769914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:25.769917 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:25.769930 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:25.769934 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:25.769981 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.770011 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.770034 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:53:25.771600 7218 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:25.773377 7218 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:25.779556 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.775453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:36771 (local address 127.0.0.1:34712)
0619 02:53:25.775816 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.775824 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.775861 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.776150 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.776153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.776155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:25.776224 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.776226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.777661 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.777662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.778543 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.778545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.779324 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.779325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.779383 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.779395 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.779450 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.779491 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260619 02:53:25.779585 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.775744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.21.164.193:36771)
0619 02:53:25.775918 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.775927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.775938 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:53:25.775982 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.775985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.776000 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.776072 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:25.776363 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.777555 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.778612 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.779246 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.779321 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.779413 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.779462 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.779498 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260619 02:53:25.779865 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.775751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.21.164.195:45845)
0619 02:53:25.776112 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.776116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.776124 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:25.776158 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.776159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.776174 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.776369 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:25.776428 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.777555 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.778839 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.779530 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.779597 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.779679 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.779751 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.779790 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":75,"threads_started":1}
I20260619 02:53:25.779834 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.775449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:45845 (local address 127.0.0.1:58806)
0619 02:53:25.775610 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.775650 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.775713 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.776267 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.776268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.776270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:25.776302 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.776303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.777631 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.777632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.778753 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.778756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.779617 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.779618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.779681 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.779685 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.779738 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.779773 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":8,"thread_start_us":102,"threads_started":1}
I20260619 02:53:25.780696 7340 tablet_service.cc:1467] Processing CreateTablet for tablet 935c3a4fb108464991c3de7d5f9d2f8f (DEFAULT_TABLE table=test-table [id=f89ad950581b44dc89abf64ce90b1eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:25.781111 7340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935c3a4fb108464991c3de7d5f9d2f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:25.781174 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.775453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:43455 (local address 127.0.0.1:52076)
0619 02:53:25.775921 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.775931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.775981 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.776237 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.776238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.776240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:25.776285 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.776287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.777474 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.777475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.778768 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.778769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.780905 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.780906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.780963 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.780966 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.781032 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.781071 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260619 02:53:25.781231 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.775939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.21.164.194:43455)
0619 02:53:25.776093 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.776097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.776106 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:25.776142 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.776144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.776157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.776225 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:25.776359 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.777406 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.779716 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.780834 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.780897 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.780979 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.781110 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.781153 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260619 02:53:25.781088 7602 tablet_service.cc:1467] Processing CreateTablet for tablet 935c3a4fb108464991c3de7d5f9d2f8f (DEFAULT_TABLE table=test-table [id=f89ad950581b44dc89abf64ce90b1eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:25.781459 7602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935c3a4fb108464991c3de7d5f9d2f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:25.781809 7471 tablet_service.cc:1467] Processing CreateTablet for tablet 935c3a4fb108464991c3de7d5f9d2f8f (DEFAULT_TABLE table=test-table [id=f89ad950581b44dc89abf64ce90b1eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:25.782073 7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935c3a4fb108464991c3de7d5f9d2f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:25.784749 7683 tablet_bootstrap.cc:492] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20: Bootstrap starting.
I20260619 02:53:25.784773 7684 tablet_bootstrap.cc:492] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: Bootstrap starting.
I20260619 02:53:25.785418 7683 tablet_bootstrap.cc:654] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:25.785418 7684 tablet_bootstrap.cc:654] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:25.785729 7684 log.cc:824] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:25.785729 7683 log.cc:824] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:25.786060 7682 tablet_bootstrap.cc:492] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7: Bootstrap starting.
I20260619 02:53:25.786499 7684 tablet_bootstrap.cc:492] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: No bootstrap required, opened a new log
I20260619 02:53:25.786572 7684 ts_tablet_manager.cc:1397] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:25.786609 7682 tablet_bootstrap.cc:654] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:25.786738 7683 tablet_bootstrap.cc:492] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20: No bootstrap required, opened a new log
I20260619 02:53:25.786906 7682 log.cc:824] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:25.787091 7683 ts_tablet_manager.cc:1397] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:53:25.788547 7684 raft_consensus.cc:348] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef305be3218444d86da6f4477102f69" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 43455 } } peers { permanent_uuid: "373ed8ea35224ce9a9036e445a0d5d20" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36771 } } peers { permanent_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45845 } }
I20260619 02:53:25.788731 7684 raft_consensus.cc:374] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:25.788789 7684 raft_consensus.cc:729] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef305be3218444d86da6f4477102f69, State: Initialized, Role: FOLLOWER
I20260619 02:53:25.789011 7684 consensus_queue.cc:260] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef305be3218444d86da6f4477102f69" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 43455 } } peers { permanent_uuid: "373ed8ea35224ce9a9036e445a0d5d20" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36771 } } peers { permanent_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45845 } }
I20260619 02:53:25.789017 7683 raft_consensus.cc:348] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef305be3218444d86da6f4477102f69" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 43455 } } peers { permanent_uuid: "373ed8ea35224ce9a9036e445a0d5d20" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36771 } } peers { permanent_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45845 } }
I20260619 02:53:25.789187 7682 tablet_bootstrap.cc:492] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7: No bootstrap required, opened a new log
I20260619 02:53:25.789206 7683 raft_consensus.cc:374] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:25.789252 7682 ts_tablet_manager.cc:1397] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 02:53:25.789255 7684 ts_tablet_manager.cc:1428] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:25.789275 7683 raft_consensus.cc:729] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 373ed8ea35224ce9a9036e445a0d5d20, State: Initialized, Role: FOLLOWER
I20260619 02:53:25.789361 7537 heartbeater.cc:503] Master 127.21.164.254:36521 was elected leader, sending a full tablet report...
I20260619 02:53:25.789451 7683 consensus_queue.cc:260] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef305be3218444d86da6f4477102f69" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 43455 } } peers { permanent_uuid: "373ed8ea35224ce9a9036e445a0d5d20" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36771 } } peers { permanent_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45845 } }
I20260619 02:53:25.789685 7683 ts_tablet_manager.cc:1428] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 02:53:25.789731 7406 heartbeater.cc:503] Master 127.21.164.254:36521 was elected leader, sending a full tablet report...
I20260619 02:53:25.791522 7682 raft_consensus.cc:348] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef305be3218444d86da6f4477102f69" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 43455 } } peers { permanent_uuid: "373ed8ea35224ce9a9036e445a0d5d20" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36771 } } peers { permanent_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45845 } }
I20260619 02:53:25.791639 7682 raft_consensus.cc:374] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:25.791671 7682 raft_consensus.cc:729] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b71ad5f19c941c490a9ad8f0b492ad7, State: Initialized, Role: FOLLOWER
I20260619 02:53:25.791782 7682 consensus_queue.cc:260] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef305be3218444d86da6f4477102f69" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 43455 } } peers { permanent_uuid: "373ed8ea35224ce9a9036e445a0d5d20" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36771 } } peers { permanent_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45845 } }
I20260619 02:53:25.791981 7682 ts_tablet_manager.cc:1428] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:25.792039 7668 heartbeater.cc:503] Master 127.21.164.254:36521 was elected leader, sending a full tablet report...
W20260619 02:53:25.888195 7538 tablet.cc:2369] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:25.940155 7689 raft_consensus.cc:482] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:25.940279 7689 raft_consensus.cc:504] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef305be3218444d86da6f4477102f69" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 43455 } } peers { permanent_uuid: "373ed8ea35224ce9a9036e445a0d5d20" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36771 } } peers { permanent_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45845 } }
I20260619 02:53:25.940685 7689 leader_election.cc:288] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ef305be3218444d86da6f4477102f69 (127.21.164.194:43455), 0b71ad5f19c941c490a9ad8f0b492ad7 (127.21.164.195:45845)
I20260619 02:53:25.943414 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.940852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:43455 (local address 127.21.164.193:52755)
0619 02:53:25.940900 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.940913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.940958 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.941186 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.941187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.941189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:25.941248 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.941251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.941930 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.941931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.942681 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.942682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.943179 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.943180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.943252 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.943255 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.943298 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.943342 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 02:53:25.943414 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.941026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:52755 (local address 127.21.164.194:43455)
0619 02:53:25.941079 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.941081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.941089 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:25.941121 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.941122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.941131 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.941178 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:25.941306 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.941862 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.942759 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.943110 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.943172 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.943180 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.943309 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.943345 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:53:25.943604 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.940859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:45845 (local address 127.21.164.193:48243)
0619 02:53:25.941056 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.941066 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:25.941114 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:25.941395 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:25.941396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:25.941398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:25.941467 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.941468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.942131 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.942132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.942853 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:25.942855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.943368 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.943369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:25.943429 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.943432 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:25.943476 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:53:25.943515 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":45,"thread_start_us":92,"threads_started":1}
I20260619 02:53:25.943595 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.940950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:48243 (local address 127.21.164.195:45845)
0619 02:53:25.941012 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.941015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.941126 (+ 111us) server_negotiation.cc:407] Connection header received
0619 02:53:25.941330 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.941331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.941340 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.941374 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:25.941579 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.942029 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.942939 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.943286 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.943346 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.943353 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.943492 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.943537 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:53:25.943794 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935c3a4fb108464991c3de7d5f9d2f8f" candidate_uuid: "373ed8ea35224ce9a9036e445a0d5d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef305be3218444d86da6f4477102f69" is_pre_election: true
I20260619 02:53:25.943956 7491 raft_consensus.cc:2454] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 373ed8ea35224ce9a9036e445a0d5d20 in term 0.
I20260619 02:53:25.944060 7622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935c3a4fb108464991c3de7d5f9d2f8f" candidate_uuid: "373ed8ea35224ce9a9036e445a0d5d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" is_pre_election: true
I20260619 02:53:25.944196 7622 raft_consensus.cc:2454] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 373ed8ea35224ce9a9036e445a0d5d20 in term 0.
I20260619 02:53:25.944188 7295 leader_election.cc:302] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 373ed8ea35224ce9a9036e445a0d5d20, 8ef305be3218444d86da6f4477102f69; no voters:
I20260619 02:53:25.944311 7689 raft_consensus.cc:2781] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:25.944355 7689 raft_consensus.cc:482] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:25.944376 7689 raft_consensus.cc:3037] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:25.945132 7689 raft_consensus.cc:504] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef305be3218444d86da6f4477102f69" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 43455 } } peers { permanent_uuid: "373ed8ea35224ce9a9036e445a0d5d20" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36771 } } peers { permanent_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45845 } }
I20260619 02:53:25.945258 7689 leader_election.cc:288] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [CANDIDATE]: Term 1 election: Requested vote from peers 8ef305be3218444d86da6f4477102f69 (127.21.164.194:43455), 0b71ad5f19c941c490a9ad8f0b492ad7 (127.21.164.195:45845)
I20260619 02:53:25.945397 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935c3a4fb108464991c3de7d5f9d2f8f" candidate_uuid: "373ed8ea35224ce9a9036e445a0d5d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef305be3218444d86da6f4477102f69"
I20260619 02:53:25.945462 7491 raft_consensus.cc:3037] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:25.945461 7622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935c3a4fb108464991c3de7d5f9d2f8f" candidate_uuid: "373ed8ea35224ce9a9036e445a0d5d20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7"
I20260619 02:53:25.945549 7622 raft_consensus.cc:3037] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:25.946153 7491 raft_consensus.cc:2454] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 373ed8ea35224ce9a9036e445a0d5d20 in term 1.
I20260619 02:53:25.946311 7295 leader_election.cc:302] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 373ed8ea35224ce9a9036e445a0d5d20, 8ef305be3218444d86da6f4477102f69; no voters:
I20260619 02:53:25.946316 7622 raft_consensus.cc:2454] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 373ed8ea35224ce9a9036e445a0d5d20 in term 1.
I20260619 02:53:25.946394 7689 raft_consensus.cc:2781] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:25.946543 7689 raft_consensus.cc:686] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [term 1 LEADER]: Becoming Leader. State: Replica: 373ed8ea35224ce9a9036e445a0d5d20, State: Running, Role: LEADER
I20260619 02:53:25.946648 7689 consensus_queue.cc:237] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef305be3218444d86da6f4477102f69" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 43455 } } peers { permanent_uuid: "373ed8ea35224ce9a9036e445a0d5d20" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36771 } } peers { permanent_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45845 } }
I20260619 02:53:25.947459 7218 catalog_manager.cc:5495] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 373ed8ea35224ce9a9036e445a0d5d20 (127.21.164.193). New cstate: current_term: 1 leader_uuid: "373ed8ea35224ce9a9036e445a0d5d20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef305be3218444d86da6f4477102f69" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 43455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "373ed8ea35224ce9a9036e445a0d5d20" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45845 } health_report { overall_health: UNKNOWN } } }
I20260619 02:53:25.960467 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.958139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.21.164.193:36771)
0619 02:53:25.958192 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:25.958194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:25.958201 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:53:25.958246 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:25.958247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:25.958265 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:25.958315 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:25.958459 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.958918 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.959718 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:25.959931 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:25.959987 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:25.960149 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:25.960259 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:25.960334 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:25.960352 (+ 18us) server_negotiation.cc:299] Negotiation successful
0619 02:53:25.960374 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 02:53:25.962387 7622 raft_consensus.cc:1261] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7 [term 1 FOLLOWER]: Refusing update from remote peer 373ed8ea35224ce9a9036e445a0d5d20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:25.962389 7491 raft_consensus.cc:1261] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69 [term 1 FOLLOWER]: Refusing update from remote peer 373ed8ea35224ce9a9036e445a0d5d20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:25.962769 7692 consensus_queue.cc:1035] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b71ad5f19c941c490a9ad8f0b492ad7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:25.962904 7689 consensus_queue.cc:1035] T 935c3a4fb108464991c3de7d5f9d2f8f P 373ed8ea35224ce9a9036e445a0d5d20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef305be3218444d86da6f4477102f69" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:25.967536 7218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:25.967661 7218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:25.968390 7218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c767abb0e2ec42678b446effee572bcf: Sending DeleteTablet for 3 replicas of tablet 935c3a4fb108464991c3de7d5f9d2f8f
I20260619 02:53:25.968729 7471 tablet_service.cc:1514] Processing DeleteTablet for tablet 935c3a4fb108464991c3de7d5f9d2f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:25 UTC) from {username='slave'} at 127.0.0.1:52076
I20260619 02:53:25.968990 7602 tablet_service.cc:1514] Processing DeleteTablet for tablet 935c3a4fb108464991c3de7d5f9d2f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:25 UTC) from {username='slave'} at 127.0.0.1:58806
I20260619 02:53:25.969074 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 7278
I20260619 02:53:25.969620 7705 tablet_replica.cc:330] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: stopping tablet replica
I20260619 02:53:25.969718 7705 raft_consensus.cc:2229] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:25.969810 7705 raft_consensus.cc:2258] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:25.970199 7705 ts_tablet_manager.cc:1905] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:25.971747 7705 ts_tablet_manager.cc:1918] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:25.971817 7705 log.cc:1196] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/935c3a4fb108464991c3de7d5f9d2f8f
I20260619 02:53:25.972054 7705 ts_tablet_manager.cc:1939] T 935c3a4fb108464991c3de7d5f9d2f8f P 8ef305be3218444d86da6f4477102f69: Deleting consensus metadata
I20260619 02:53:25.972470 7204 catalog_manager.cc:4841] TS 8ef305be3218444d86da6f4477102f69 (127.21.164.194:43455): tablet 935c3a4fb108464991c3de7d5f9d2f8f (table test-table [id=f89ad950581b44dc89abf64ce90b1eb8]) successfully deleted
I20260619 02:53:25.972759 7706 tablet_replica.cc:330] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7: stopping tablet replica
I20260619 02:53:25.972870 7706 raft_consensus.cc:2229] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:25.972980 7706 raft_consensus.cc:2258] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:25.973356 7706 ts_tablet_manager.cc:1905] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:53:25.974730 7202 connection.cc:535] client connection to 127.21.164.193:36771 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 02:53:25.974825 7202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 02:53:25.974890 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 7409
I20260619 02:53:25.975235 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:25.975129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:36771 (local address 127.0.0.1:34732)
0619 02:53:25.975173 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:25.975205 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:36771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 02:53:25.975328 7202 catalog_manager.cc:4584] TS 373ed8ea35224ce9a9036e445a0d5d20 (127.21.164.193:36771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 935c3a4fb108464991c3de7d5f9d2f8f: Network error: Client connection negotiation failed: client connection to 127.21.164.193:36771: connect: Connection refused (error 111)
I20260619 02:53:25.975440 7706 ts_tablet_manager.cc:1918] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:25.975512 7706 log.cc:1196] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/935c3a4fb108464991c3de7d5f9d2f8f
I20260619 02:53:25.975731 7706 ts_tablet_manager.cc:1939] T 935c3a4fb108464991c3de7d5f9d2f8f P 0b71ad5f19c941c490a9ad8f0b492ad7: Deleting consensus metadata
I20260619 02:53:25.976109 7205 catalog_manager.cc:4841] TS 0b71ad5f19c941c490a9ad8f0b492ad7 (127.21.164.195:45845): tablet 935c3a4fb108464991c3de7d5f9d2f8f (table test-table [id=f89ad950581b44dc89abf64ce90b1eb8]) successfully deleted
I20260619 02:53:25.979394 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 7540
I20260619 02:53:25.983173 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 7187
2026-06-19T02:53:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (812 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:26 dist-test-slave-jt0t krb5kdc[7710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:26 dist-test-slave-jt0t krb5kdc[7710](info): set up 2 sockets
Jun 19 02:53:26 dist-test-slave-jt0t krb5kdc[7710](info): commencing operation
krb5kdc: starting...
W20260619 02:53:26.944418 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:27 dist-test-slave-jt0t krb5kdc[7710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:53:27.060946 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42549
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:42549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:27.141717 7723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:27.141999 7723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:27.142026 7723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:27.143208 7723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:27.143250 7723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:27.143263 7723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:27.143275 7723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:27.143285 7723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:27.144752 7723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:42549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:27.144943 7723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:27.145121 7723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:27.149734 7731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:27.149746 7729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:27.149781 7723 server_base.cc:1034] running on GCE node
W20260619 02:53:27.149753 7728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:27.150207 7723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:27.150461 7723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:27.151598 7723 hybrid_clock.cc:648] HybridClock initialized: now 1781837607151586 us; error 30 us; skew 500 ppm
Jun 19 02:53:27 dist-test-slave-jt0t krb5kdc[7710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:27.153992 7723 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:27.154481 7723 webserver.cc:460] Webserver started at http://127.21.164.254:43637/ using document root <none> and password file <none>
I20260619 02:53:27.154685 7723 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:27.154729 7723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:27.154834 7723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:27.155862 7723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "2c23c3d5663541d48e6704220422a024"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "5b9cc770f55487ce14dfdd2ba0b8ba8b"
server_key_iv: "1c6cbebfe3335d6537e9eade419e7d90"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.156183 7723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "2c23c3d5663541d48e6704220422a024"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "5b9cc770f55487ce14dfdd2ba0b8ba8b"
server_key_iv: "1c6cbebfe3335d6537e9eade419e7d90"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.157505 7723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:27.158212 7738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:27.158396 7723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:27.158475 7723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "2c23c3d5663541d48e6704220422a024"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "5b9cc770f55487ce14dfdd2ba0b8ba8b"
server_key_iv: "1c6cbebfe3335d6537e9eade419e7d90"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.158533 7723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:27.168522 7723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:27.170483 7723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:27.170593 7723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:27.174240 7723 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:42549
I20260619 02:53:27.174278 7790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:42549 every 8 connection(s)
I20260619 02:53:27.174602 7723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:27.175463 7791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:27.175848 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 7723
I20260619 02:53:27.175925 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:27.176098 22163 external_mini_cluster.cc:1428] Setting key 71b6ed5adf7eade43ef5f7018a9290a1
I20260619 02:53:27.178274 7791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024: Bootstrap starting.
I20260619 02:53:27.178855 7791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:27.179116 7791 log.cc:824] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:27.179895 7791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024: No bootstrap required, opened a new log
I20260619 02:53:27.182034 7791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c23c3d5663541d48e6704220422a024" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42549 } }
I20260619 02:53:27.182217 7791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:27.182256 7791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c23c3d5663541d48e6704220422a024, State: Initialized, Role: FOLLOWER
I20260619 02:53:27.182390 7791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c23c3d5663541d48e6704220422a024" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42549 } }
I20260619 02:53:27.182466 7791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:27.182497 7791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:27.182545 7791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:27.183336 7791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c23c3d5663541d48e6704220422a024" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42549 } }
I20260619 02:53:27.183452 7791 leader_election.cc:302] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c23c3d5663541d48e6704220422a024; no voters:
I20260619 02:53:27.183662 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:27.183624 7791 leader_election.cc:288] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:27.183804 7796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [term 1 LEADER]: Becoming Leader. State: Replica: 2c23c3d5663541d48e6704220422a024, State: Running, Role: LEADER
I20260619 02:53:27.183961 7796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c23c3d5663541d48e6704220422a024" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42549 } }
I20260619 02:53:27.184020 7791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:27.184309 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c23c3d5663541d48e6704220422a024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c23c3d5663541d48e6704220422a024" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42549 } } }
I20260619 02:53:27.184546 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:27.184473 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c23c3d5663541d48e6704220422a024. Latest consensus state: current_term: 1 leader_uuid: "2c23c3d5663541d48e6704220422a024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c23c3d5663541d48e6704220422a024" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42549 } } }
I20260619 02:53:27.184742 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024 [sys.catalog]: This master's current role is: LEADER
Jun 19 02:53:27 dist-test-slave-jt0t krb5kdc[7710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:27.185133 7806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:27.185642 7806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:27.187222 7806 catalog_manager.cc:1347] Generated new cluster ID: 3de69d7ed9ef4463bf538e5267f34502
I20260619 02:53:27.187695 7806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:27.189146 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.176607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54136 (local address 127.21.164.254:42549)
0619 02:53:27.178562 (+ 1955us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:27.178570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:27.178601 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:53:27.179102 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:27.179112 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:27.179167 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:27.179329 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:27.180062 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.181025 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.182109 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.182658 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.182751 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.185657 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:27.185662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:27.185662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:27.185676 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:27.187476 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:27.187862 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:27.187866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:27.187866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:27.187972 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:27.188172 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:27.188172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:27.188173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:27.188487 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:27.188631 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:27.188776 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 02:53:27.188922 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1907,"thread_start_us":95,"threads_started":1}
I20260619 02:53:27.197513 7806 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:27.198235 7806 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:27.204823 7806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024: Generated new TSK 0
I20260619 02:53:27.205022 7806 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:53:27.233244 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42549
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:27.313338 7817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:27.313561 7817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:27.313596 7817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:27.314786 7817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:27.314832 7817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:27.314869 7817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:53:27.316337 7817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:27.316576 7817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:27.316829 7817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:27.321933 7822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:27.321934 7825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:27.321952 7817 server_base.cc:1034] running on GCE node
W20260619 02:53:27.321933 7823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:27.322372 7817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:27.322609 7817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:27.323758 7817 hybrid_clock.cc:648] HybridClock initialized: now 1781837607323739 us; error 33 us; skew 500 ppm
Jun 19 02:53:27 dist-test-slave-jt0t krb5kdc[7710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:27.326220 7817 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:53:27.326699 7817 webserver.cc:460] Webserver started at http://127.21.164.193:41585/ using document root <none> and password file <none>
I20260619 02:53:27.326898 7817 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:27.326936 7817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:27.327028 7817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:27.328023 7817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "97d90712d61c4528a784ea6a23b06fa5"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "2b90f9430467b4d4713b763e739d53bf"
server_key_iv: "8e1c3680bf9dcd7ad3df584d19c56149"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.328337 7817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "97d90712d61c4528a784ea6a23b06fa5"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "2b90f9430467b4d4713b763e739d53bf"
server_key_iv: "8e1c3680bf9dcd7ad3df584d19c56149"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.329696 7817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:53:27.330468 7832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:27.330655 7817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:53:27.330739 7817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "97d90712d61c4528a784ea6a23b06fa5"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "2b90f9430467b4d4713b763e739d53bf"
server_key_iv: "8e1c3680bf9dcd7ad3df584d19c56149"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.330804 7817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:27.365387 7817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:27.367586 7817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:27.367733 7817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:27.367960 7817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:27.368264 7817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:27.368297 7817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:27.368327 7817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:27.368345 7817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:27.373867 7817 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:44149
I20260619 02:53:27.373914 7945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:44149 every 8 connection(s)
I20260619 02:53:27.374241 7817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:27.378556 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 7817
I20260619 02:53:27.378681 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:27.378871 22163 external_mini_cluster.cc:1428] Setting key 01bad3692e4d9efe5b115c1459b77995
Jun 19 02:53:27 dist-test-slave-jt0t krb5kdc[7710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:27.383759 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.374992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:33531 (local address 127.21.164.254:42549)
0619 02:53:27.375051 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:27.375053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:27.375239 (+ 186us) server_negotiation.cc:407] Connection header received
0619 02:53:27.375746 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:27.375748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:27.375761 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:27.375817 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:27.376798 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.377297 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.378307 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.378604 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.378679 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.380897 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:27.380898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:27.380898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:27.380905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:27.382295 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:27.382720 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:27.382721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:27.382721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:27.382787 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:27.383091 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:27.383092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:27.383093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:27.383224 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:27.383285 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:27.383552 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 02:53:27.383665 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:53:27.383962 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.374949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42549 (local address 127.21.164.193:33531)
0619 02:53:27.375119 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:27.375158 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:27.375645 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:27.375897 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:27.375900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:27.376472 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:27.376700 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.376705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.377379 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.377380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.378219 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.378222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.378708 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.378710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.378862 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.379370 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:27.379378 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:27.380792 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:27.382392 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:27.382398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:27.382401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:27.382645 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:27.382881 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:27.382882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:27.382882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:27.382997 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:27.383289 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:27.383296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:27.383434 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 02:53:27.383524 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:53:27.383685 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260619 02:53:27.384429 7946 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42549
I20260619 02:53:27.384541 7946 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:27.384769 7946 heartbeater.cc:511] Master 127.21.164.254:42549 requested a full tablet report, sending...
I20260619 02:53:27.385319 7755 ts_manager.cc:194] Registered new tserver with Master: 97d90712d61c4528a784ea6a23b06fa5 (127.21.164.193:44149)
I20260619 02:53:27.386036 7755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:33531
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:53:27.403317 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42549
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:27.482025 7951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:27.482236 7951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:27.482268 7951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:27.483398 7951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:27.483439 7951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:27.483474 7951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:53:27.485039 7951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:27.485268 7951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:27.485471 7951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:27.490203 7957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:27.490268 7951 server_base.cc:1034] running on GCE node
W20260619 02:53:27.490273 7959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:27.490203 7956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:27.490600 7951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:27.490813 7951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:27.491971 7951 hybrid_clock.cc:648] HybridClock initialized: now 1781837607491952 us; error 31 us; skew 500 ppm
Jun 19 02:53:27 dist-test-slave-jt0t krb5kdc[7710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:27.494360 7951 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:53:27.494769 7951 webserver.cc:460] Webserver started at http://127.21.164.194:35837/ using document root <none> and password file <none>
I20260619 02:53:27.494957 7951 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:27.495003 7951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:27.495123 7951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:27.496119 7951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "180a8f54b03e48dcba1a37b75bb6b4f3"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "7a86a17d1ec020a3566832537b67923f"
server_key_iv: "9896dc3664b2a3e983a8715175275b2a"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.496431 7951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "180a8f54b03e48dcba1a37b75bb6b4f3"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "7a86a17d1ec020a3566832537b67923f"
server_key_iv: "9896dc3664b2a3e983a8715175275b2a"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.497702 7951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:27.498404 7966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:27.498566 7951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:27.498642 7951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "180a8f54b03e48dcba1a37b75bb6b4f3"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "7a86a17d1ec020a3566832537b67923f"
server_key_iv: "9896dc3664b2a3e983a8715175275b2a"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.498708 7951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:27.523321 7951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:27.525388 7951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:27.525523 7951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:27.525741 7951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:27.526027 7951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:27.526062 7951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:27.526091 7951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:27.526115 7951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:27.531337 7951 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:38273
I20260619 02:53:27.531383 8079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:38273 every 8 connection(s)
I20260619 02:53:27.531684 7951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
Jun 19 02:53:27 dist-test-slave-jt0t krb5kdc[7710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:27.538141 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 7951
I20260619 02:53:27.538259 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:27.538453 22163 external_mini_cluster.cc:1428] Setting key 50ac8b5734ea0a897c421879514db815
I20260619 02:53:27.540827 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.532348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:54419 (local address 127.21.164.254:42549)
0619 02:53:27.532413 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:27.532415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:27.532601 (+ 186us) server_negotiation.cc:407] Connection header received
0619 02:53:27.533171 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:27.533172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:27.533184 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:27.533231 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:27.534218 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.534780 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.535734 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.535920 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.535976 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.538021 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:27.538023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:27.538023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:27.538031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:27.539460 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:27.539856 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:27.539857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:27.539857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:27.539914 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:27.540148 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:27.540148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:27.540149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:27.540275 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:27.540339 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:27.540699 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 02:53:27.540761 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:53:27.541100 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.532314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42549 (local address 127.21.164.194:54419)
0619 02:53:27.532484 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:27.532520 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:27.533071 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:27.533315 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:27.533318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:27.533923 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:27.534127 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.534133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.534867 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.534867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.535658 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.535660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.535985 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.535985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.536120 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.536563 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:27.536571 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:27.537934 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:27.539559 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:27.539566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:27.539568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:27.539792 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:27.539998 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:27.539999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:27.539999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:27.540097 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:27.540362 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:27.540370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:27.540543 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 02:53:27.540687 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 02:53:27.540860 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260619 02:53:27.541584 8080 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42549
I20260619 02:53:27.541675 8080 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:27.541847 8080 heartbeater.cc:511] Master 127.21.164.254:42549 requested a full tablet report, sending...
I20260619 02:53:27.542217 7755 ts_manager.cc:194] Registered new tserver with Master: 180a8f54b03e48dcba1a37b75bb6b4f3 (127.21.164.194:38273)
I20260619 02:53:27.542605 7755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:54419
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:53:27.563074 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42549
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:27.643115 8085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:27.643324 8085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:27.643348 8085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:27.644490 8085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:27.644527 8085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:27.644554 8085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:53:27.646051 8085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:42549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:27.646250 8085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:27.646446 8085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:27.651369 8091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:27.651361 8093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:27.651361 8090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:27.651386 8085 server_base.cc:1034] running on GCE node
I20260619 02:53:27.651736 8085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:27.651965 8085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:27.653115 8085 hybrid_clock.cc:648] HybridClock initialized: now 1781837607653097 us; error 26 us; skew 500 ppm
Jun 19 02:53:27 dist-test-slave-jt0t krb5kdc[7710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:27.655397 8085 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:53:27.655814 8085 webserver.cc:460] Webserver started at http://127.21.164.195:43473/ using document root <none> and password file <none>
I20260619 02:53:27.656031 8085 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:27.656080 8085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:27.656191 8085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:27.657262 8085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "42c29779e0724aac857d9326bd0d9932"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "4b901943eaf9ae7969336edca74d3f75"
server_key_iv: "2747a92dba87db3b0ec131e0addef60c"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.657595 8085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "42c29779e0724aac857d9326bd0d9932"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "4b901943eaf9ae7969336edca74d3f75"
server_key_iv: "2747a92dba87db3b0ec131e0addef60c"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.658802 8085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:27.659487 8100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:27.659646 8085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:27.659725 8085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "42c29779e0724aac857d9326bd0d9932"
format_stamp: "Formatted at 2026-06-19 02:53:27 on dist-test-slave-jt0t"
server_key: "4b901943eaf9ae7969336edca74d3f75"
server_key_iv: "2747a92dba87db3b0ec131e0addef60c"
server_key_version: "encryptionkey@0"
I20260619 02:53:27.659783 8085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:27.671712 8085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:27.673628 8085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:27.673764 8085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:27.673991 8085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:27.674284 8085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:27.674317 8085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:27.674346 8085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:27.674364 8085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:27.679749 8085 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:41825
I20260619 02:53:27.679800 8213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:41825 every 8 connection(s)
I20260619 02:53:27.680094 8085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:53:27 dist-test-slave-jt0t krb5kdc[7710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837607, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:27.687834 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 8085
I20260619 02:53:27.687958 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:27.688150 22163 external_mini_cluster.cc:1428] Setting key 61ba3369c0d38453431944f68d67155f
I20260619 02:53:27.689417 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.680768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:33643 (local address 127.21.164.254:42549)
0619 02:53:27.680827 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:27.680829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:27.681074 (+ 245us) server_negotiation.cc:407] Connection header received
0619 02:53:27.681509 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:27.681511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:27.681521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:27.681572 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:27.682461 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.682994 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.684020 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.684247 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.684314 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.686563 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:27.686564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:27.686565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:27.686572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:27.687875 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:27.688328 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:27.688329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:27.688329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:27.688397 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:27.688712 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:27.688713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:27.688713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:27.688863 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:27.688921 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:27.689244 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 02:53:27.689330 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:53:27.689580 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.680777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:42549 (local address 127.21.164.195:33643)
0619 02:53:27.680919 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:27.680955 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:27.681411 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:27.681629 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:27.681632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:27.682172 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:27.682365 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.682370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.683065 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.683065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.683922 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.683924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.684336 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.684337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.684483 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.684947 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:27.684955 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:27.686475 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:27.687965 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:27.687972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:27.687975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:27.688256 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:27.688476 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:27.688477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:27.688477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:27.688624 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:27.688931 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:27.688936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:27.689100 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 02:53:27.689193 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:53:27.689353 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260619 02:53:27.689966 8214 heartbeater.cc:347] Connected to a master server at 127.21.164.254:42549
I20260619 02:53:27.690049 8214 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:27.690196 8214 heartbeater.cc:511] Master 127.21.164.254:42549 requested a full tablet report, sending...
I20260619 02:53:27.690501 7755 ts_manager.cc:194] Registered new tserver with Master: 42c29779e0724aac857d9326bd0d9932 (127.21.164.195:41825)
I20260619 02:53:27.690841 7755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:33643
I20260619 02:53:27.699888 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:27.706846 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.701218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.21.164.254:42549)
0619 02:53:27.701267 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:27.701269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:27.701372 (+ 103us) server_negotiation.cc:407] Connection header received
0619 02:53:27.701424 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:27.701425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:27.701433 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:27.701480 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:27.702000 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.702517 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.703200 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.703381 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.703430 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.704538 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:27.704539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:27.704539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:27.704547 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:27.705873 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:27.706232 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:27.706232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:27.706233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:27.706280 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:27.706420 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:27.706420 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:27.706421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:27.706536 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:27.706617 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:27.706712 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 02:53:27.706760 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:53:27.708328 7755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:27.710256 7755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:27.715950 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.712390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:38273 (local address 127.0.0.1:35828)
0619 02:53:27.712701 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:27.712745 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:27.712813 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:27.713133 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:27.713134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:27.713136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:27.713181 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.713182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.714078 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.714079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.714883 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.714885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.715659 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.715660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.715738 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.715752 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 02:53:27.715819 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 02:53:27.715872 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":45,"threads_started":1}
I20260619 02:53:27.715971 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.712499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.21.164.194:38273)
0619 02:53:27.712696 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:27.712702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:27.712894 (+ 192us) server_negotiation.cc:407] Connection header received
0619 02:53:27.712936 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:27.712938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:27.712954 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:27.713024 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:27.713251 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.713990 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.714962 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.715585 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.715649 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.715705 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:27.715839 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 02:53:27.715885 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260619 02:53:27.716825 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.712401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:44149 (local address 127.0.0.1:57974)
0619 02:53:27.712785 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:27.712795 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:27.712851 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:27.713025 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:27.713028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:27.713030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:27.713114 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.713116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.714505 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.714506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.715718 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.715719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.716565 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.716566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.716629 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.716633 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:27.716713 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:53:27.716753 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260619 02:53:27.716840 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.712510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.21.164.193:44149)
0619 02:53:27.712769 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:27.712772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:27.712838 (+ 66us) server_negotiation.cc:407] Connection header received
0619 02:53:27.712920 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:27.712923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:27.712937 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:27.713003 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:27.713204 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.714395 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.715774 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.716487 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.716551 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.716604 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:27.716727 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 02:53:27.716765 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":73,"threads_started":1}
I20260619 02:53:27.716909 8014 tablet_service.cc:1467] Processing CreateTablet for tablet 92015f3710bd45049d885f7b024ddd9b (DEFAULT_TABLE table=test-table [id=491ae5e123384a7395429ec00eeb6696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:27.717168 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.712601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:41825 (local address 127.0.0.1:46896)
0619 02:53:27.712966 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:27.712978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:27.713016 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:27.713340 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:27.713341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:27.713343 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:27.713377 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.713378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.714605 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.714606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.715678 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:27.715680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.716809 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.716810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:27.716877 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.716881 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:27.717062 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 02:53:27.717105 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":17,"thread_start_us":86,"threads_started":1}
I20260619 02:53:27.717303 8014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92015f3710bd45049d885f7b024ddd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:27.717657 7880 tablet_service.cc:1467] Processing CreateTablet for tablet 92015f3710bd45049d885f7b024ddd9b (DEFAULT_TABLE table=test-table [id=491ae5e123384a7395429ec00eeb6696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:27.717974 7880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92015f3710bd45049d885f7b024ddd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:27.720633 8231 tablet_bootstrap.cc:492] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: Bootstrap starting.
I20260619 02:53:27.721123 8232 tablet_bootstrap.cc:492] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5: Bootstrap starting.
I20260619 02:53:27.721282 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:27.712813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.21.164.195:41825)
0619 02:53:27.713144 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:27.713148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:27.713157 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:27.713193 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:27.713195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:27.713209 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:27.713274 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:27.713447 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.714502 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.715766 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:27.716642 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:27.716726 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:27.716775 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:27.721150 (+ 4375us) server_negotiation.cc:299] Negotiation successful
0619 02:53:27.721204 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":75,"threads_started":1}
I20260619 02:53:27.721305 8231 tablet_bootstrap.cc:654] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:27.721706 8232 tablet_bootstrap.cc:654] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:27.721850 8231 log.cc:824] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:27.721978 8148 tablet_service.cc:1467] Processing CreateTablet for tablet 92015f3710bd45049d885f7b024ddd9b (DEFAULT_TABLE table=test-table [id=491ae5e123384a7395429ec00eeb6696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:27.722113 8232 log.cc:824] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:27.722259 8148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92015f3710bd45049d885f7b024ddd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:27.722625 8231 tablet_bootstrap.cc:492] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: No bootstrap required, opened a new log
I20260619 02:53:27.722680 8231 ts_tablet_manager.cc:1397] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 02:53:27.724530 8231 raft_consensus.cc:348] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 38273 } } peers { permanent_uuid: "97d90712d61c4528a784ea6a23b06fa5" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44149 } } peers { permanent_uuid: "42c29779e0724aac857d9326bd0d9932" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41825 } }
I20260619 02:53:27.724741 8231 raft_consensus.cc:374] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:27.724778 8231 raft_consensus.cc:729] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180a8f54b03e48dcba1a37b75bb6b4f3, State: Initialized, Role: FOLLOWER
I20260619 02:53:27.724905 8231 consensus_queue.cc:260] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 38273 } } peers { permanent_uuid: "97d90712d61c4528a784ea6a23b06fa5" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44149 } } peers { permanent_uuid: "42c29779e0724aac857d9326bd0d9932" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41825 } }
I20260619 02:53:27.725199 8231 ts_tablet_manager.cc:1428] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 02:53:27.725217 8080 heartbeater.cc:503] Master 127.21.164.254:42549 was elected leader, sending a full tablet report...
I20260619 02:53:27.725353 8236 tablet_bootstrap.cc:492] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: Bootstrap starting.
I20260619 02:53:27.725940 8232 tablet_bootstrap.cc:492] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5: No bootstrap required, opened a new log
I20260619 02:53:27.725960 8236 tablet_bootstrap.cc:654] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:27.726164 8232 ts_tablet_manager.cc:1397] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 02:53:27.726444 8236 log.cc:824] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:27.727131 8236 tablet_bootstrap.cc:492] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: No bootstrap required, opened a new log
I20260619 02:53:27.727200 8236 ts_tablet_manager.cc:1397] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:27.728107 8232 raft_consensus.cc:348] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 38273 } } peers { permanent_uuid: "97d90712d61c4528a784ea6a23b06fa5" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44149 } } peers { permanent_uuid: "42c29779e0724aac857d9326bd0d9932" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41825 } }
I20260619 02:53:27.728289 8232 raft_consensus.cc:374] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:27.728330 8232 raft_consensus.cc:729] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d90712d61c4528a784ea6a23b06fa5, State: Initialized, Role: FOLLOWER
I20260619 02:53:27.728446 8232 consensus_queue.cc:260] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 38273 } } peers { permanent_uuid: "97d90712d61c4528a784ea6a23b06fa5" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44149 } } peers { permanent_uuid: "42c29779e0724aac857d9326bd0d9932" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41825 } }
I20260619 02:53:27.728755 7946 heartbeater.cc:503] Master 127.21.164.254:42549 was elected leader, sending a full tablet report...
I20260619 02:53:27.729028 8232 ts_tablet_manager.cc:1428] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:27.729039 8236 raft_consensus.cc:348] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 38273 } } peers { permanent_uuid: "97d90712d61c4528a784ea6a23b06fa5" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44149 } } peers { permanent_uuid: "42c29779e0724aac857d9326bd0d9932" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41825 } }
I20260619 02:53:27.729192 8236 raft_consensus.cc:374] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:27.729225 8236 raft_consensus.cc:729] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c29779e0724aac857d9326bd0d9932, State: Initialized, Role: FOLLOWER
I20260619 02:53:27.729331 8236 consensus_queue.cc:260] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 38273 } } peers { permanent_uuid: "97d90712d61c4528a784ea6a23b06fa5" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44149 } } peers { permanent_uuid: "42c29779e0724aac857d9326bd0d9932" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41825 } }
I20260619 02:53:27.729658 8236 ts_tablet_manager.cc:1428] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 02:53:27.729696 8214 heartbeater.cc:503] Master 127.21.164.254:42549 was elected leader, sending a full tablet report...
W20260619 02:53:27.782269 8081 tablet.cc:2369] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:53:27.875047 7947 tablet.cc:2369] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:53:27.930619 8215 tablet.cc:2369] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:28.082468 8239 raft_consensus.cc:482] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:28.082609 8239 raft_consensus.cc:504] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 38273 } } peers { permanent_uuid: "97d90712d61c4528a784ea6a23b06fa5" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44149 } } peers { permanent_uuid: "42c29779e0724aac857d9326bd0d9932" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41825 } }
I20260619 02:53:28.082983 8239 leader_election.cc:288] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 180a8f54b03e48dcba1a37b75bb6b4f3 (127.21.164.194:38273), 97d90712d61c4528a784ea6a23b06fa5 (127.21.164.193:44149)
I20260619 02:53:28.086216 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.083285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:35389 (local address 127.21.164.193:44149)
0619 02:53:28.083342 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.083344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.083539 (+ 195us) server_negotiation.cc:407] Connection header received
0619 02:53:28.083586 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.083587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.083597 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.083636 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:28.083757 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.084499 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.085559 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.085915 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.085977 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.085986 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.086116 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.086152 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:53:28.086229 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.083223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:44149 (local address 127.21.164.195:35389)
0619 02:53:28.083482 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.083490 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:28.083535 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:28.083649 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:28.083650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:28.083651 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:28.083708 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.083709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.084574 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.084575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.085476 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.085478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.085983 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.085984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.086054 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.086058 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:28.086101 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 02:53:28.086140 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":93,"thread_start_us":108,"threads_started":1}
I20260619 02:53:28.086318 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.083216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:38273 (local address 127.21.164.195:59311)
0619 02:53:28.083285 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.083296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:28.083339 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:28.083506 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:28.083507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:28.083509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:28.083594 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.083596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.084527 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.084529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.085511 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.085514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.086053 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.086054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.086142 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.086144 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:53:28.086188 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:53:28.086225 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":27}
I20260619 02:53:28.086316 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.083315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:59311 (local address 127.21.164.194:38273)
0619 02:53:28.083364 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.083366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.083374 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:28.083404 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.083405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.083416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.083484 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:28.083653 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.084418 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.085614 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.085979 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.086041 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.086049 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.086203 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.086241 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:53:28.086639 8034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92015f3710bd45049d885f7b024ddd9b" candidate_uuid: "42c29779e0724aac857d9326bd0d9932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" is_pre_election: true
I20260619 02:53:28.086640 7900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92015f3710bd45049d885f7b024ddd9b" candidate_uuid: "42c29779e0724aac857d9326bd0d9932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d90712d61c4528a784ea6a23b06fa5" is_pre_election: true
I20260619 02:53:28.086803 8034 raft_consensus.cc:2454] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42c29779e0724aac857d9326bd0d9932 in term 0.
I20260619 02:53:28.086803 7900 raft_consensus.cc:2454] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42c29779e0724aac857d9326bd0d9932 in term 0.
I20260619 02:53:28.087024 8102 leader_election.cc:302] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42c29779e0724aac857d9326bd0d9932, 97d90712d61c4528a784ea6a23b06fa5; no voters:
I20260619 02:53:28.087201 8239 raft_consensus.cc:2781] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:28.087249 8239 raft_consensus.cc:482] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:28.087277 8239 raft_consensus.cc:3037] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:28.087946 8239 raft_consensus.cc:504] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 38273 } } peers { permanent_uuid: "97d90712d61c4528a784ea6a23b06fa5" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44149 } } peers { permanent_uuid: "42c29779e0724aac857d9326bd0d9932" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41825 } }
I20260619 02:53:28.088063 8239 leader_election.cc:288] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [CANDIDATE]: Term 1 election: Requested vote from peers 180a8f54b03e48dcba1a37b75bb6b4f3 (127.21.164.194:38273), 97d90712d61c4528a784ea6a23b06fa5 (127.21.164.193:44149)
I20260619 02:53:28.088240 8034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92015f3710bd45049d885f7b024ddd9b" candidate_uuid: "42c29779e0724aac857d9326bd0d9932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3"
I20260619 02:53:28.088256 7900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92015f3710bd45049d885f7b024ddd9b" candidate_uuid: "42c29779e0724aac857d9326bd0d9932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d90712d61c4528a784ea6a23b06fa5"
I20260619 02:53:28.088321 8034 raft_consensus.cc:3037] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:28.088321 7900 raft_consensus.cc:3037] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:28.089223 7900 raft_consensus.cc:2454] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42c29779e0724aac857d9326bd0d9932 in term 1.
I20260619 02:53:28.089223 8034 raft_consensus.cc:2454] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42c29779e0724aac857d9326bd0d9932 in term 1.
I20260619 02:53:28.089427 8103 leader_election.cc:302] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 180a8f54b03e48dcba1a37b75bb6b4f3, 42c29779e0724aac857d9326bd0d9932; no voters:
I20260619 02:53:28.089514 8239 raft_consensus.cc:2781] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:28.089663 8239 raft_consensus.cc:686] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 1 LEADER]: Becoming Leader. State: Replica: 42c29779e0724aac857d9326bd0d9932, State: Running, Role: LEADER
I20260619 02:53:28.089751 8239 consensus_queue.cc:237] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 38273 } } peers { permanent_uuid: "97d90712d61c4528a784ea6a23b06fa5" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44149 } } peers { permanent_uuid: "42c29779e0724aac857d9326bd0d9932" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41825 } }
I20260619 02:53:28.090570 7755 catalog_manager.cc:5495] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42c29779e0724aac857d9326bd0d9932 (127.21.164.195). New cstate: current_term: 1 leader_uuid: "42c29779e0724aac857d9326bd0d9932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 38273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97d90712d61c4528a784ea6a23b06fa5" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42c29779e0724aac857d9326bd0d9932" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 41825 } health_report { overall_health: HEALTHY } } }
I20260619 02:53:28.159752 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.157390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.21.164.195:41825)
0619 02:53:28.157454 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.157456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.157465 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:28.157528 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.157528 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.157547 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.157608 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:28.157760 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.158249 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.158995 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.159189 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.159242 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.159406 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:28.159496 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:28.159578 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.159651 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.159663 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:53:28.161434 8034 raft_consensus.cc:1261] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3 [term 1 FOLLOWER]: Refusing update from remote peer 42c29779e0724aac857d9326bd0d9932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:28.161429 7900 raft_consensus.cc:1261] T 92015f3710bd45049d885f7b024ddd9b P 97d90712d61c4528a784ea6a23b06fa5 [term 1 FOLLOWER]: Refusing update from remote peer 42c29779e0724aac857d9326bd0d9932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:28.161731 8239 consensus_queue.cc:1035] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "180a8f54b03e48dcba1a37b75bb6b4f3" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 38273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:28.161859 8241 consensus_queue.cc:1035] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97d90712d61c4528a784ea6a23b06fa5" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:28.164121 8244 mvcc.cc:205] Tried to move back new op lower bound from 7298406843026583552 to 7298406842736177152. Current Snapshot: MvccSnapshot[applied={T|T < 7298406843026583552}]
I20260619 02:53:28.164924 8246 mvcc.cc:205] Tried to move back new op lower bound from 7298406843026583552 to 7298406842736177152. Current Snapshot: MvccSnapshot[applied={T|T < 7298406843026583552 or (T in {7298406843026583552})}]
I20260619 02:53:28.165033 8245 mvcc.cc:205] Tried to move back new op lower bound from 7298406843026583552 to 7298406842736177152. Current Snapshot: MvccSnapshot[applied={T|T < 7298406843026583552}]
I20260619 02:53:28.166396 7755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:28.166496 7755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:28.167222 7755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c23c3d5663541d48e6704220422a024: Sending DeleteTablet for 3 replicas of tablet 92015f3710bd45049d885f7b024ddd9b
I20260619 02:53:28.167579 8148 tablet_service.cc:1514] Processing DeleteTablet for tablet 92015f3710bd45049d885f7b024ddd9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:28 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:46896
I20260619 02:53:28.167596 8014 tablet_service.cc:1514] Processing DeleteTablet for tablet 92015f3710bd45049d885f7b024ddd9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:28 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:35828
I20260619 02:53:28.167824 7880 tablet_service.cc:1514] Processing DeleteTablet for tablet 92015f3710bd45049d885f7b024ddd9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:28 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:57974
I20260619 02:53:28.167919 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 7817
I20260619 02:53:28.168617 8254 tablet_replica.cc:330] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: stopping tablet replica
I20260619 02:53:28.168737 8254 raft_consensus.cc:2229] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:53:28.168865 8254 raft_consensus.cc:2258] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:28.169329 8254 ts_tablet_manager.cc:1905] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:28.170967 8254 ts_tablet_manager.cc:1918] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:28.171034 8254 log.cc:1196] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/92015f3710bd45049d885f7b024ddd9b
I20260619 02:53:28.171255 8254 ts_tablet_manager.cc:1939] T 92015f3710bd45049d885f7b024ddd9b P 42c29779e0724aac857d9326bd0d9932: Deleting consensus metadata
I20260619 02:53:28.171599 7740 catalog_manager.cc:4841] TS 42c29779e0724aac857d9326bd0d9932 (127.21.164.195:41825): tablet 92015f3710bd45049d885f7b024ddd9b (table test-table [id=491ae5e123384a7395429ec00eeb6696]) successfully deleted
I20260619 02:53:28.172777 8255 tablet_replica.cc:330] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: stopping tablet replica
I20260619 02:53:28.172897 8255 raft_consensus.cc:2229] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:28.173031 8255 raft_consensus.cc:2258] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:28.173489 8255 ts_tablet_manager.cc:1905] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:28.175364 8255 ts_tablet_manager.cc:1918] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:28.175486 8255 log.cc:1196] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/92015f3710bd45049d885f7b024ddd9b
I20260619 02:53:28.175762 8255 ts_tablet_manager.cc:1939] T 92015f3710bd45049d885f7b024ddd9b P 180a8f54b03e48dcba1a37b75bb6b4f3: Deleting consensus metadata
I20260619 02:53:28.176241 7741 catalog_manager.cc:4841] TS 180a8f54b03e48dcba1a37b75bb6b4f3 (127.21.164.194:38273): tablet 92015f3710bd45049d885f7b024ddd9b (table test-table [id=491ae5e123384a7395429ec00eeb6696]) successfully deleted
W20260619 02:53:28.176997 7740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:44149 (error 108)
I20260619 02:53:28.177407 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 7951
I20260619 02:53:28.178016 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.177557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:44149 (local address 127.0.0.1:57976)
0619 02:53:28.177938 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.177982 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:44149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357}
W20260619 02:53:28.179438 7740 catalog_manager.cc:4584] TS 97d90712d61c4528a784ea6a23b06fa5 (127.21.164.193:44149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92015f3710bd45049d885f7b024ddd9b: Network error: Client connection negotiation failed: client connection to 127.21.164.193:44149: connect: Connection refused (error 111)
I20260619 02:53:28.182391 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 8085
I20260619 02:53:28.186585 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 7723
2026-06-19T02:53:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2204 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T02:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:28Z Disabled control of system clock
I20260619 02:53:28.211977 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45041
--webserver_interface=127.21.164.254
--webserver_port=0
--builtin_ntp_servers=127.21.164.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:45041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:28.305658 8261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:28.305872 8261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:28.305892 8261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:28.307047 8261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:28.307090 8261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:28.307101 8261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:28.307113 8261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:28.307123 8261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:28.308465 8261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:45041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:28.308718 8261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:28.309015 8261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:28.313735 8267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:28.313741 8269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:28.314028 8266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:28.314103 8261 server_base.cc:1034] running on GCE node
I20260619 02:53:28.314361 8261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:28.314659 8261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:28.315809 8261 hybrid_clock.cc:648] HybridClock initialized: now 1781837608315793 us; error 30 us; skew 500 ppm
I20260619 02:53:28.317055 8261 webserver.cc:460] Webserver started at http://127.21.164.254:35337/ using document root <none> and password file <none>
I20260619 02:53:28.317293 8261 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:28.317338 8261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:28.317467 8261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:28.318526 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "112d0071d10c4af8a861da8f79324b0f"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "2136d0eacd744d9de5e688deb450a4dc"
server_key_iv: "b69088ae96e0d2451f67134e3ea6d86a"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.318856 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "112d0071d10c4af8a861da8f79324b0f"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "2136d0eacd744d9de5e688deb450a4dc"
server_key_iv: "b69088ae96e0d2451f67134e3ea6d86a"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.320037 8261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:28.320721 8275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:28.320881 8261 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:28.320961 8261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "112d0071d10c4af8a861da8f79324b0f"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "2136d0eacd744d9de5e688deb450a4dc"
server_key_iv: "b69088ae96e0d2451f67134e3ea6d86a"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.321031 8261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:28.328909 8261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:28.329221 8261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:28.329345 8261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:28.333071 8261 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45041
I20260619 02:53:28.333127 8327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45041 every 8 connection(s)
I20260619 02:53:28.333451 8261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:28.334398 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:28.337046 8328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f: Bootstrap starting.
I20260619 02:53:28.337671 8328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:28.337903 8328 log.cc:824] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:28.338522 8328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f: No bootstrap required, opened a new log
I20260619 02:53:28.338814 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 8261
I20260619 02:53:28.338925 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:28.339161 22163 external_mini_cluster.cc:1428] Setting key 0b1cfac0e75e67b7cfcca2f49e7a8ef6
I20260619 02:53:28.340574 8328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112d0071d10c4af8a861da8f79324b0f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45041 } }
I20260619 02:53:28.340780 8328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:28.340817 8328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 112d0071d10c4af8a861da8f79324b0f, State: Initialized, Role: FOLLOWER
I20260619 02:53:28.340945 8328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112d0071d10c4af8a861da8f79324b0f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45041 } }
I20260619 02:53:28.341025 8328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:28.341070 8328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:28.341118 8328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:28.341820 8328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112d0071d10c4af8a861da8f79324b0f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45041 } }
I20260619 02:53:28.341929 8328 leader_election.cc:302] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 112d0071d10c4af8a861da8f79324b0f; no voters:
I20260619 02:53:28.342097 8328 leader_election.cc:288] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:28.342152 8333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:28.342303 8333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [term 1 LEADER]: Becoming Leader. State: Replica: 112d0071d10c4af8a861da8f79324b0f, State: Running, Role: LEADER
I20260619 02:53:28.342339 8328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:28.342391 8333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112d0071d10c4af8a861da8f79324b0f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45041 } }
I20260619 02:53:28.342887 8335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 112d0071d10c4af8a861da8f79324b0f. Latest consensus state: current_term: 1 leader_uuid: "112d0071d10c4af8a861da8f79324b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112d0071d10c4af8a861da8f79324b0f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45041 } } }
I20260619 02:53:28.342885 8334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "112d0071d10c4af8a861da8f79324b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "112d0071d10c4af8a861da8f79324b0f" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45041 } } }
I20260619 02:53:28.343000 8335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:28.343041 8334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:28.343783 8340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:28.344255 8340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:28.345973 8340 catalog_manager.cc:1347] Generated new cluster ID: fea6a463e0df4e17b7578eae793c571f
I20260619 02:53:28.346020 8340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:28.351854 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.339654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.21.164.254:45041)
0619 02:53:28.339881 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.339893 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.339927 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:53:28.340531 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.340540 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.340629 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.340797 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:28.340990 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.342697 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.343849 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.344580 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.344720 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.351073 (+ 6353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:28.351078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:28.351079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:28.351317 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:28.351365 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:28.351382 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:28.351459 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.351520 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.351615 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260619 02:53:28.362344 8340 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:28.362859 8340 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:28.367100 8340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f: Generated new TSK 0
I20260619 02:53:28.367331 8340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:28.373703 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45041
--builtin_ntp_servers=127.21.164.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:28.453441 8352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:28.453670 8352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:28.453701 8352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:28.454864 8352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:28.454905 8352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:28.454939 8352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:53:28.456307 8352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:28.456562 8352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:28.456881 8352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:28.462738 8358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:28.462713 8357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:28.462839 8360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:28.463104 8352 server_base.cc:1034] running on GCE node
I20260619 02:53:28.463356 8352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:28.463620 8352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:28.464773 8352 hybrid_clock.cc:648] HybridClock initialized: now 1781837608464753 us; error 40 us; skew 500 ppm
I20260619 02:53:28.466264 8352 webserver.cc:460] Webserver started at http://127.21.164.193:43565/ using document root <none> and password file <none>
I20260619 02:53:28.466554 8352 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:28.466620 8352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:28.466745 8352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:28.468107 8352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "d0e88cef2a8b4edfa865044c7faef94e"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "1326b071c7b3ebd3e8eb9347bea72f98"
server_key_iv: "59d72f59e3daadbc82edbc7a5fbd0eee"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.468510 8352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "d0e88cef2a8b4edfa865044c7faef94e"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "1326b071c7b3ebd3e8eb9347bea72f98"
server_key_iv: "59d72f59e3daadbc82edbc7a5fbd0eee"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.470039 8352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:53:28.470791 8366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:28.470930 8352 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:53:28.471019 8352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "d0e88cef2a8b4edfa865044c7faef94e"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "1326b071c7b3ebd3e8eb9347bea72f98"
server_key_iv: "59d72f59e3daadbc82edbc7a5fbd0eee"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.471088 8352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:28.481789 8352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:28.482075 8352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:28.482196 8352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:28.482389 8352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:28.482683 8352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:28.482715 8352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:28.482743 8352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:28.482766 8352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:28.488489 8352 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:36847
I20260619 02:53:28.488543 8479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:36847 every 8 connection(s)
I20260619 02:53:28.488900 8352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:28.493788 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.489590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:53745 (local address 127.21.164.254:45041)
0619 02:53:28.489651 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.489653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.489883 (+ 230us) server_negotiation.cc:407] Connection header received
0619 02:53:28.490378 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.490379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.490392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.490453 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:28.491071 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.491561 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.492499 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.492734 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.492783 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.493568 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:28.493569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:28.493569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:28.493574 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:28.493590 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:28.493593 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:28.493635 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.493692 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.493719 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 02:53:28.493948 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.489579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45041 (local address 127.21.164.193:53745)
0619 02:53:28.489763 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.489804 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:28.490282 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:28.490513 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:28.490516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:28.490524 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:28.491000 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.491004 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.491622 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.491622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.492416 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.492418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.492805 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.492806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.492919 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.493431 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:28.493432 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:28.493461 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:28.493486 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:28.493637 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:28.493639 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:28.493641 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:53:28.493685 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:53:28.493763 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260619 02:53:28.494405 8480 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45041
I20260619 02:53:28.494486 8480 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:28.494632 8480 heartbeater.cc:511] Master 127.21.164.254:45041 requested a full tablet report, sending...
I20260619 02:53:28.495113 8292 ts_manager.cc:194] Registered new tserver with Master: d0e88cef2a8b4edfa865044c7faef94e (127.21.164.193:36847)
I20260619 02:53:28.495728 8292 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.193:53745
I20260619 02:53:28.498677 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 8352
I20260619 02:53:28.498780 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:28.498956 22163 external_mini_cluster.cc:1428] Setting key 390c9a5bed99c1f9c2c1b96d948d05b2
I20260619 02:53:28.499797 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45041
--builtin_ntp_servers=127.21.164.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:28.582707 8483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:28.582914 8483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:28.582935 8483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:28.584125 8483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:28.584170 8483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:28.584194 8483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:53:28.585575 8483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:28.585812 8483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:28.586115 8483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:53:28.590950 8483 server_base.cc:1034] running on GCE node
W20260619 02:53:28.590888 8491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:28.590896 8489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:28.590925 8488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:28.591288 8483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:28.591516 8483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:28.592686 8483 hybrid_clock.cc:648] HybridClock initialized: now 1781837608592671 us; error 25 us; skew 500 ppm
I20260619 02:53:28.593999 8483 webserver.cc:460] Webserver started at http://127.21.164.194:40907/ using document root <none> and password file <none>
I20260619 02:53:28.594246 8483 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:28.594296 8483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:28.594424 8483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:28.595510 8483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "259f7e3f35bc49c49e933899d4463039"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "4df80eaacd88a605e5348720ec069dfd"
server_key_iv: "389e87085fe09b2c18987e1f70bc9a3b"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.595839 8483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "259f7e3f35bc49c49e933899d4463039"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "4df80eaacd88a605e5348720ec069dfd"
server_key_iv: "389e87085fe09b2c18987e1f70bc9a3b"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.597150 8483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:28.597826 8497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:28.597971 8483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:28.598049 8483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "259f7e3f35bc49c49e933899d4463039"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "4df80eaacd88a605e5348720ec069dfd"
server_key_iv: "389e87085fe09b2c18987e1f70bc9a3b"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.598109 8483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:28.625797 8483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:28.626134 8483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:28.626271 8483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:28.626498 8483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:28.626817 8483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:28.626852 8483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:28.626883 8483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:28.626904 8483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:28.632232 8483 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:41211
I20260619 02:53:28.632294 8610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:41211 every 8 connection(s)
I20260619 02:53:28.632613 8483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:28.634572 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 8483
I20260619 02:53:28.634671 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:28.634845 22163 external_mini_cluster.cc:1428] Setting key 67d22480e7a28c2fcf1ead0ac62cb7d7
I20260619 02:53:28.635829 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45041
--builtin_ntp_servers=127.21.164.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:53:28.638232 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.633415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:56559 (local address 127.21.164.254:45041)
0619 02:53:28.633487 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.633489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.633697 (+ 208us) server_negotiation.cc:407] Connection header received
0619 02:53:28.634192 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.634193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.634204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.634252 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:28.635002 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.635677 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.636880 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.637085 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.637143 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.637966 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:28.637967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:28.637967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:28.637973 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:28.637991 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:28.637995 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:28.638039 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.638118 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.638150 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:53:28.638441 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.633375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45041 (local address 127.21.164.194:56559)
0619 02:53:28.633581 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.633616 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:28.634102 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:28.634315 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:28.634319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:28.634327 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:28.634903 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.634912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.635785 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.635785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.636794 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.636797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.637157 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.637158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.637292 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.637818 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:28.637819 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:28.637851 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:28.637881 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:28.638048 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:28.638054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:28.638057 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:28.638118 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 02:53:28.638231 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260619 02:53:28.638885 8611 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45041
I20260619 02:53:28.638993 8611 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:28.639213 8611 heartbeater.cc:511] Master 127.21.164.254:45041 requested a full tablet report, sending...
I20260619 02:53:28.639611 8292 ts_manager.cc:194] Registered new tserver with Master: 259f7e3f35bc49c49e933899d4463039 (127.21.164.194:41211)
I20260619 02:53:28.640105 8292 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.194:56559
W20260619 02:53:28.715700 8614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:28.715909 8614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:28.715930 8614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:28.717083 8614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:28.717118 8614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:28.717141 8614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:53:28.718426 8614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:45041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:28.718634 8614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:28.718880 8614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:53:28.723773 8614 server_base.cc:1034] running on GCE node
W20260619 02:53:28.723764 8620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:28.723755 8619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:28.723749 8622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:28.724154 8614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:28.724388 8614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:28.725518 8614 hybrid_clock.cc:648] HybridClock initialized: now 1781837608725518 us; error 36 us; skew 500 ppm
I20260619 02:53:28.726722 8614 webserver.cc:460] Webserver started at http://127.21.164.195:43503/ using document root <none> and password file <none>
I20260619 02:53:28.726941 8614 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:28.726989 8614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:28.727118 8614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:28.728159 8614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "862f06e0f3624548928d51a0a99e4ebf"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "0bd2325b35665603327de353239bfe7c"
server_key_iv: "cd6ade4c0e81b241d7543ac3daaaae43"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.728484 8614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "862f06e0f3624548928d51a0a99e4ebf"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "0bd2325b35665603327de353239bfe7c"
server_key_iv: "cd6ade4c0e81b241d7543ac3daaaae43"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.729745 8614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:28.730522 8628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:28.730681 8614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:28.730762 8614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "862f06e0f3624548928d51a0a99e4ebf"
format_stamp: "Formatted at 2026-06-19 02:53:28 on dist-test-slave-jt0t"
server_key: "0bd2325b35665603327de353239bfe7c"
server_key_iv: "cd6ade4c0e81b241d7543ac3daaaae43"
server_key_version: "encryptionkey@0"
I20260619 02:53:28.730829 8614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:28.754658 8614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:28.754961 8614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:28.755084 8614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:28.755303 8614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:28.755623 8614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:28.755656 8614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:28.755686 8614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:28.755710 8614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:28.761070 8614 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:40401
I20260619 02:53:28.761116 8741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:40401 every 8 connection(s)
I20260619 02:53:28.761427 8614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:53:28.766548 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.762097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:34883 (local address 127.21.164.254:45041)
0619 02:53:28.762159 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.762161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.762349 (+ 188us) server_negotiation.cc:407] Connection header received
0619 02:53:28.762849 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.762851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.762863 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.762924 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:28.763618 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.764132 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.765264 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.765460 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.765525 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.766322 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:28.766324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:28.766324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:28.766328 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:28.766344 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:28.766348 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:28.766390 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.766455 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.766486 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:53:28.766743 8744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.762063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:45041 (local address 127.21.164.195:34883)
0619 02:53:28.762231 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.762270 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:28.762741 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:28.762985 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:28.762989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:28.762997 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:28.763517 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.763522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.764226 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.764227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.765169 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.765171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.765539 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.765540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.765654 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.766175 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:28.766176 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:28.766207 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:28.766232 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:28.766397 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:28.766400 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:28.766403 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:28.766448 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 02:53:28.766531 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260619 02:53:28.767230 8742 heartbeater.cc:347] Connected to a master server at 127.21.164.254:45041
I20260619 02:53:28.767320 8742 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:28.767498 8742 heartbeater.cc:511] Master 127.21.164.254:45041 requested a full tablet report, sending...
I20260619 02:53:28.767851 8292 ts_manager.cc:194] Registered new tserver with Master: 862f06e0f3624548928d51a0a99e4ebf (127.21.164.195:40401)
I20260619 02:53:28.768219 8292 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.195:34883
I20260619 02:53:28.770644 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 8614
I20260619 02:53:28.770737 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:28.770915 22163 external_mini_cluster.cc:1428] Setting key 21f818711f4c7c291857c97909b1d456
I20260619 02:53:28.771819 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:28.776144 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.773171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.21.164.254:45041)
0619 02:53:28.773228 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.773230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.773310 (+ 80us) server_negotiation.cc:407] Connection header received
0619 02:53:28.773360 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.773361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.773371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.773420 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:28.773584 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.774096 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.774850 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.775123 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.775181 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.775843 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:28.775845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:28.775845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:28.775850 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:28.775867 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:28.775872 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:28.775951 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.776036 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.776055 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:53:28.777490 8292 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:28.778981 8292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:28.785006 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.781235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.21.164.195:40401)
0619 02:53:28.781387 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.781391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.781401 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:53:28.781440 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.781442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.781460 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.781881 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:28.781935 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.782963 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.784070 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.784601 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.784703 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.784795 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.784878 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.784921 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260619 02:53:28.784993 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.780971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:40401 (local address 127.0.0.1:38094)
0619 02:53:28.781089 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.781127 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:28.781207 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:28.781565 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:28.781569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:28.781571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:28.781642 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.781644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.783063 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.783065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.783985 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.783988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.784701 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.784702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.784781 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.784793 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 02:53:28.784855 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 02:53:28.784909 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":50,"thread_start_us":72,"threads_started":1}
I20260619 02:53:28.785283 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.781063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.21.164.194:41211)
0619 02:53:28.781203 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.781207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.781384 (+ 177us) server_negotiation.cc:407] Connection header received
0619 02:53:28.781505 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.781507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.781522 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.781571 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:28.781807 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.782992 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.784230 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.784928 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.784990 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.785063 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.785162 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.785198 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260619 02:53:28.785476 8753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.780974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:41211 (local address 127.0.0.1:59296)
0619 02:53:28.781313 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.781323 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:28.781412 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:28.781664 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:28.781665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:28.781667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:28.781712 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.781714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.783075 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.783076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.784156 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.784159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.785001 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.785002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.785066 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.785070 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:28.785364 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 02:53:28.785411 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":21,"thread_start_us":124,"threads_started":1}
I20260619 02:53:28.785800 8676 tablet_service.cc:1467] Processing CreateTablet for tablet 62e75b663ed84611a85bcf1beb47c215 (DEFAULT_TABLE table=test-table [id=ca38327c83a646f89ca9e6a8f995019c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:28.786175 8676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e75b663ed84611a85bcf1beb47c215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:28.786124 8545 tablet_service.cc:1467] Processing CreateTablet for tablet 62e75b663ed84611a85bcf1beb47c215 (DEFAULT_TABLE table=test-table [id=ca38327c83a646f89ca9e6a8f995019c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:28.786413 8545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e75b663ed84611a85bcf1beb47c215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:28.789538 8756 tablet_bootstrap.cc:492] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf: Bootstrap starting.
I20260619 02:53:28.789788 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.781192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:36847 (local address 127.0.0.1:51716)
0619 02:53:28.781397 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.781405 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:28.781444 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:28.781731 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:28.781732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:28.781733 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:28.781761 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.781762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.782661 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.782662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.783857 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.783860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.789535 (+ 5675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.789536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.789618 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.789623 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:53:28.789683 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:53:28.789728 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260619 02:53:28.790024 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.781391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.21.164.193:36847)
0619 02:53:28.781583 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.781586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.781594 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:28.781630 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.781632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.781654 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.781722 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:28.781836 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.782580 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.783925 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.784599 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.784710 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.784795 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.789892 (+ 5097us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.789958 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20260619 02:53:28.790246 8756 tablet_bootstrap.cc:654] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:28.790498 8756 log.cc:824] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:28.790599 8414 tablet_service.cc:1467] Processing CreateTablet for tablet 62e75b663ed84611a85bcf1beb47c215 (DEFAULT_TABLE table=test-table [id=ca38327c83a646f89ca9e6a8f995019c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:28.790820 8757 tablet_bootstrap.cc:492] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039: Bootstrap starting.
I20260619 02:53:28.790917 8414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e75b663ed84611a85bcf1beb47c215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:28.791234 8756 tablet_bootstrap.cc:492] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf: No bootstrap required, opened a new log
I20260619 02:53:28.791301 8756 ts_tablet_manager.cc:1397] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:28.791421 8757 tablet_bootstrap.cc:654] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:28.791720 8757 log.cc:824] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:28.792673 8757 tablet_bootstrap.cc:492] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039: No bootstrap required, opened a new log
I20260619 02:53:28.792742 8757 ts_tablet_manager.cc:1397] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:28.793732 8760 tablet_bootstrap.cc:492] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e: Bootstrap starting.
I20260619 02:53:28.793829 8756 raft_consensus.cc:348] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f06e0f3624548928d51a0a99e4ebf" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 40401 } } peers { permanent_uuid: "259f7e3f35bc49c49e933899d4463039" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41211 } } peers { permanent_uuid: "d0e88cef2a8b4edfa865044c7faef94e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36847 } }
I20260619 02:53:28.793998 8756 raft_consensus.cc:374] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:28.794037 8756 raft_consensus.cc:729] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862f06e0f3624548928d51a0a99e4ebf, State: Initialized, Role: FOLLOWER
I20260619 02:53:28.794175 8756 consensus_queue.cc:260] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f06e0f3624548928d51a0a99e4ebf" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 40401 } } peers { permanent_uuid: "259f7e3f35bc49c49e933899d4463039" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41211 } } peers { permanent_uuid: "d0e88cef2a8b4edfa865044c7faef94e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36847 } }
I20260619 02:53:28.794303 8760 tablet_bootstrap.cc:654] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:28.794441 8756 ts_tablet_manager.cc:1428] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:28.794574 8742 heartbeater.cc:503] Master 127.21.164.254:45041 was elected leader, sending a full tablet report...
I20260619 02:53:28.794595 8760 log.cc:824] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:28.795100 8757 raft_consensus.cc:348] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f06e0f3624548928d51a0a99e4ebf" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 40401 } } peers { permanent_uuid: "259f7e3f35bc49c49e933899d4463039" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41211 } } peers { permanent_uuid: "d0e88cef2a8b4edfa865044c7faef94e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36847 } }
I20260619 02:53:28.795217 8757 raft_consensus.cc:374] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:28.795246 8757 raft_consensus.cc:729] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259f7e3f35bc49c49e933899d4463039, State: Initialized, Role: FOLLOWER
I20260619 02:53:28.795324 8760 tablet_bootstrap.cc:492] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e: No bootstrap required, opened a new log
I20260619 02:53:28.795354 8757 consensus_queue.cc:260] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f06e0f3624548928d51a0a99e4ebf" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 40401 } } peers { permanent_uuid: "259f7e3f35bc49c49e933899d4463039" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41211 } } peers { permanent_uuid: "d0e88cef2a8b4edfa865044c7faef94e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36847 } }
I20260619 02:53:28.795393 8760 ts_tablet_manager.cc:1397] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:28.795586 8757 ts_tablet_manager.cc:1428] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:28.795626 8611 heartbeater.cc:503] Master 127.21.164.254:45041 was elected leader, sending a full tablet report...
I20260619 02:53:28.797443 8760 raft_consensus.cc:348] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f06e0f3624548928d51a0a99e4ebf" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 40401 } } peers { permanent_uuid: "259f7e3f35bc49c49e933899d4463039" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41211 } } peers { permanent_uuid: "d0e88cef2a8b4edfa865044c7faef94e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36847 } }
I20260619 02:53:28.797569 8760 raft_consensus.cc:374] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:28.797590 8760 raft_consensus.cc:729] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e88cef2a8b4edfa865044c7faef94e, State: Initialized, Role: FOLLOWER
I20260619 02:53:28.797694 8760 consensus_queue.cc:260] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f06e0f3624548928d51a0a99e4ebf" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 40401 } } peers { permanent_uuid: "259f7e3f35bc49c49e933899d4463039" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41211 } } peers { permanent_uuid: "d0e88cef2a8b4edfa865044c7faef94e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36847 } }
I20260619 02:53:28.797904 8760 ts_tablet_manager.cc:1428] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 02:53:28.797967 8480 heartbeater.cc:503] Master 127.21.164.254:45041 was elected leader, sending a full tablet report...
I20260619 02:53:28.836854 8764 raft_consensus.cc:482] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:28.836990 8764 raft_consensus.cc:504] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f06e0f3624548928d51a0a99e4ebf" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 40401 } } peers { permanent_uuid: "259f7e3f35bc49c49e933899d4463039" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41211 } } peers { permanent_uuid: "d0e88cef2a8b4edfa865044c7faef94e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36847 } }
I20260619 02:53:28.837335 8764 leader_election.cc:288] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 862f06e0f3624548928d51a0a99e4ebf (127.21.164.195:40401), 259f7e3f35bc49c49e933899d4463039 (127.21.164.194:41211)
I20260619 02:53:28.840732 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.837591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:36293 (local address 127.21.164.194:41211)
0619 02:53:28.837634 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.837636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.837648 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:53:28.837721 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.837723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.837733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.837802 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:28.837966 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.838599 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.840037 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.840392 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.840456 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.840465 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.840599 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.840639 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:53:28.840754 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.837532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:41211 (local address 127.21.164.193:36293)
0619 02:53:28.837594 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.837605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:28.837650 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:28.837829 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:28.837829 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:28.837831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:28.837905 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.837907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.838663 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.838664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.839917 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.839920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.840470 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.840471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.840538 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.840541 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:28.840586 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 02:53:28.840634 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":30}
I20260619 02:53:28.840763 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.837590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:56949 (local address 127.21.164.195:40401)
0619 02:53:28.837649 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.837651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.837798 (+ 147us) server_negotiation.cc:407] Connection header received
0619 02:53:28.837864 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.837865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.837876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.837933 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:28.838063 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.838594 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.840037 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.840394 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.840470 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.840479 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.840637 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.840700 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:53:28.840984 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.837541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:40401 (local address 127.21.164.193:56949)
0619 02:53:28.837722 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.837733 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:28.837794 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:28.837936 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:28.837936 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:28.837938 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:28.838005 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.838006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.838676 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.838677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.839919 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:28.839920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.840501 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.840502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:28.840572 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.840575 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:28.840621 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:53:28.840681 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":34,"thread_start_us":90,"threads_started":1}
I20260619 02:53:28.841115 8565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e75b663ed84611a85bcf1beb47c215" candidate_uuid: "d0e88cef2a8b4edfa865044c7faef94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259f7e3f35bc49c49e933899d4463039" is_pre_election: true
I20260619 02:53:28.841297 8565 raft_consensus.cc:2454] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e88cef2a8b4edfa865044c7faef94e in term 0.
I20260619 02:53:28.841353 8696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e75b663ed84611a85bcf1beb47c215" candidate_uuid: "d0e88cef2a8b4edfa865044c7faef94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862f06e0f3624548928d51a0a99e4ebf" is_pre_election: true
I20260619 02:53:28.841490 8696 raft_consensus.cc:2454] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e88cef2a8b4edfa865044c7faef94e in term 0.
I20260619 02:53:28.841531 8367 leader_election.cc:302] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259f7e3f35bc49c49e933899d4463039, d0e88cef2a8b4edfa865044c7faef94e; no voters:
I20260619 02:53:28.841663 8764 raft_consensus.cc:2781] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:28.841733 8764 raft_consensus.cc:482] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:28.841758 8764 raft_consensus.cc:3037] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:28.842517 8764 raft_consensus.cc:504] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f06e0f3624548928d51a0a99e4ebf" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 40401 } } peers { permanent_uuid: "259f7e3f35bc49c49e933899d4463039" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41211 } } peers { permanent_uuid: "d0e88cef2a8b4edfa865044c7faef94e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36847 } }
I20260619 02:53:28.842654 8764 leader_election.cc:288] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [CANDIDATE]: Term 1 election: Requested vote from peers 862f06e0f3624548928d51a0a99e4ebf (127.21.164.195:40401), 259f7e3f35bc49c49e933899d4463039 (127.21.164.194:41211)
I20260619 02:53:28.842796 8565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e75b663ed84611a85bcf1beb47c215" candidate_uuid: "d0e88cef2a8b4edfa865044c7faef94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259f7e3f35bc49c49e933899d4463039"
I20260619 02:53:28.842823 8696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e75b663ed84611a85bcf1beb47c215" candidate_uuid: "d0e88cef2a8b4edfa865044c7faef94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862f06e0f3624548928d51a0a99e4ebf"
I20260619 02:53:28.842878 8565 raft_consensus.cc:3037] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:28.842895 8696 raft_consensus.cc:3037] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:28.843677 8696 raft_consensus.cc:2454] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e88cef2a8b4edfa865044c7faef94e in term 1.
I20260619 02:53:28.843715 8565 raft_consensus.cc:2454] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e88cef2a8b4edfa865044c7faef94e in term 1.
I20260619 02:53:28.843845 8368 leader_election.cc:302] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862f06e0f3624548928d51a0a99e4ebf, d0e88cef2a8b4edfa865044c7faef94e; no voters:
I20260619 02:53:28.843950 8764 raft_consensus.cc:2781] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:28.844097 8764 raft_consensus.cc:686] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 1 LEADER]: Becoming Leader. State: Replica: d0e88cef2a8b4edfa865044c7faef94e, State: Running, Role: LEADER
I20260619 02:53:28.844192 8764 consensus_queue.cc:237] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f06e0f3624548928d51a0a99e4ebf" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 40401 } } peers { permanent_uuid: "259f7e3f35bc49c49e933899d4463039" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41211 } } peers { permanent_uuid: "d0e88cef2a8b4edfa865044c7faef94e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36847 } }
I20260619 02:53:28.844991 8292 catalog_manager.cc:5495] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e reported cstate change: term changed from 0 to 1, leader changed from <none> to d0e88cef2a8b4edfa865044c7faef94e (127.21.164.193). New cstate: current_term: 1 leader_uuid: "d0e88cef2a8b4edfa865044c7faef94e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862f06e0f3624548928d51a0a99e4ebf" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 40401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259f7e3f35bc49c49e933899d4463039" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e88cef2a8b4edfa865044c7faef94e" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 36847 } health_report { overall_health: HEALTHY } } }
I20260619 02:53:28.857712 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.855412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.21.164.193:36847)
0619 02:53:28.855473 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:28.855475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:28.855481 (+ 6us) server_negotiation.cc:407] Connection header received
0619 02:53:28.855516 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:28.855517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:28.855535 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:28.855593 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:28.855694 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.856156 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.856910 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:28.857113 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:28.857166 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:28.857339 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:28.857425 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:28.857507 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:28.857600 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:53:28.857610 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:53:28.859452 8565 raft_consensus.cc:1261] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039 [term 1 FOLLOWER]: Refusing update from remote peer d0e88cef2a8b4edfa865044c7faef94e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:28.859452 8696 raft_consensus.cc:1261] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf [term 1 FOLLOWER]: Refusing update from remote peer d0e88cef2a8b4edfa865044c7faef94e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:28.859797 8766 consensus_queue.cc:1035] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [LEADER]: Connected to new peer: Peer: permanent_uuid: "862f06e0f3624548928d51a0a99e4ebf" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 40401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:28.859920 8764 consensus_queue.cc:1035] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [LEADER]: Connected to new peer: Peer: permanent_uuid: "259f7e3f35bc49c49e933899d4463039" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:28.862313 8770 mvcc.cc:205] Tried to move back new op lower bound from 7298406845885865984 to 7298406845826408448. Current Snapshot: MvccSnapshot[applied={T|T < 7298406845885865984}]
I20260619 02:53:28.862689 8771 mvcc.cc:205] Tried to move back new op lower bound from 7298406845885865984 to 7298406845826408448. Current Snapshot: MvccSnapshot[applied={T|T < 7298406845885865984}]
I20260619 02:53:28.864672 8291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:28.864774 8291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:28.865533 8291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 112d0071d10c4af8a861da8f79324b0f: Sending DeleteTablet for 3 replicas of tablet 62e75b663ed84611a85bcf1beb47c215
I20260619 02:53:28.865922 8676 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e75b663ed84611a85bcf1beb47c215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:28 UTC) from {username='slave'} at 127.0.0.1:38094
I20260619 02:53:28.865921 8545 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e75b663ed84611a85bcf1beb47c215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:28 UTC) from {username='slave'} at 127.0.0.1:59296
I20260619 02:53:28.865980 8414 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e75b663ed84611a85bcf1beb47c215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:28 UTC) from {username='slave'} at 127.0.0.1:51716
I20260619 02:53:28.866184 8781 tablet_replica.cc:330] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e: stopping tablet replica
I20260619 02:53:28.866278 8781 raft_consensus.cc:2229] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:53:28.866341 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 8352
I20260619 02:53:28.866374 8781 raft_consensus.cc:2258] T 62e75b663ed84611a85bcf1beb47c215 P d0e88cef2a8b4edfa865044c7faef94e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:28.866888 8780 tablet_replica.cc:330] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039: stopping tablet replica
I20260619 02:53:28.866993 8780 raft_consensus.cc:2229] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:28.867108 8780 raft_consensus.cc:2258] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:28.867280 8779 tablet_replica.cc:330] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf: stopping tablet replica
I20260619 02:53:28.867380 8779 raft_consensus.cc:2229] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:28.867472 8779 raft_consensus.cc:2258] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:28.867727 8780 ts_tablet_manager.cc:1905] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:28.868013 8779 ts_tablet_manager.cc:1905] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:28.869475 8780 ts_tablet_manager.cc:1918] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:28.869554 8780 log.cc:1196] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/62e75b663ed84611a85bcf1beb47c215
I20260619 02:53:28.869547 8779 ts_tablet_manager.cc:1918] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:28.869599 8779 log.cc:1196] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/62e75b663ed84611a85bcf1beb47c215
I20260619 02:53:28.869797 8779 ts_tablet_manager.cc:1939] T 62e75b663ed84611a85bcf1beb47c215 P 862f06e0f3624548928d51a0a99e4ebf: Deleting consensus metadata
I20260619 02:53:28.869805 8780 ts_tablet_manager.cc:1939] T 62e75b663ed84611a85bcf1beb47c215 P 259f7e3f35bc49c49e933899d4463039: Deleting consensus metadata
I20260619 02:53:28.870219 8277 catalog_manager.cc:4841] TS 862f06e0f3624548928d51a0a99e4ebf (127.21.164.195:40401): tablet 62e75b663ed84611a85bcf1beb47c215 (table test-table [id=ca38327c83a646f89ca9e6a8f995019c]) successfully deleted
I20260619 02:53:28.870291 8276 catalog_manager.cc:4841] TS 259f7e3f35bc49c49e933899d4463039 (127.21.164.194:41211): tablet 62e75b663ed84611a85bcf1beb47c215 (table test-table [id=ca38327c83a646f89ca9e6a8f995019c]) successfully deleted
W20260619 02:53:28.871918 8277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:36847 (error 108)
I20260619 02:53:28.872360 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 8483
I20260619 02:53:28.872380 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:28.872251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:36847 (local address 127.0.0.1:51726)
0619 02:53:28.872309 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:28.872351 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 02:53:28.872879 8277 catalog_manager.cc:4584] TS d0e88cef2a8b4edfa865044c7faef94e (127.21.164.193:36847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62e75b663ed84611a85bcf1beb47c215: Network error: Client connection negotiation failed: client connection to 127.21.164.193:36847: connect: Connection refused (error 111)
I20260619 02:53:28.876092 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 8614
I20260619 02:53:28.879956 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 8261
2026-06-19T02:53:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (692 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:28 dist-test-slave-jt0t krb5kdc[8785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:28 dist-test-slave-jt0t krb5kdc[8785](info): set up 2 sockets
Jun 19 02:53:28 dist-test-slave-jt0t krb5kdc[8785](info): commencing operation
krb5kdc: starting...
W20260619 02:53:29.828269 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:29 dist-test-slave-jt0t krb5kdc[8785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:53:29.939906 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:39897
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:39897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:30.019495 8798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:30.019718 8798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:30.019742 8798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:30.020922 8798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:30.020967 8798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:30.020980 8798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:30.020991 8798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:30.021001 8798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:30.022514 8798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:39897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:30.022784 8798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:30.022987 8798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:30.027830 8806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:30.027837 8803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:30.028003 8798 server_base.cc:1034] running on GCE node
W20260619 02:53:30.027881 8804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:30.028334 8798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:30.028618 8798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:30.029788 8798 hybrid_clock.cc:648] HybridClock initialized: now 1781837610029775 us; error 28 us; skew 500 ppm
Jun 19 02:53:30 dist-test-slave-jt0t krb5kdc[8785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837610, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:30.032239 8798 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:30.032724 8798 webserver.cc:460] Webserver started at http://127.21.164.254:33085/ using document root <none> and password file <none>
I20260619 02:53:30.032948 8798 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:30.032996 8798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:30.033129 8798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:30.034147 8798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "ba92d7677fde4a65b2e8c9ad1c69da57"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "53327f232cf7382f9d7807f40d249d5c"
server_key_iv: "cff64b1ec9e6c9e49725731b4a47a6e3"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.034474 8798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "ba92d7677fde4a65b2e8c9ad1c69da57"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "53327f232cf7382f9d7807f40d249d5c"
server_key_iv: "cff64b1ec9e6c9e49725731b4a47a6e3"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.035691 8798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:53:30.036391 8813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.036543 8798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:53:30.036618 8798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "ba92d7677fde4a65b2e8c9ad1c69da57"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "53327f232cf7382f9d7807f40d249d5c"
server_key_iv: "cff64b1ec9e6c9e49725731b4a47a6e3"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.036710 8798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:30.081336 8798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:30.083328 8798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:30.083472 8798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:30.087175 8798 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:39897
I20260619 02:53:30.087214 8865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:39897 every 8 connection(s)
I20260619 02:53:30.087558 8798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:30.088359 8866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:30.090744 8866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57: Bootstrap starting.
I20260619 02:53:30.091305 8866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:30.091567 8866 log.cc:824] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:30.092226 8866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57: No bootstrap required, opened a new log
I20260619 02:53:30.093833 8866 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba92d7677fde4a65b2e8c9ad1c69da57" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39897 } }
I20260619 02:53:30.093973 8866 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:30.093997 8866 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba92d7677fde4a65b2e8c9ad1c69da57, State: Initialized, Role: FOLLOWER
I20260619 02:53:30.094125 8866 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba92d7677fde4a65b2e8c9ad1c69da57" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39897 } }
I20260619 02:53:30.094194 8866 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:30.094219 8866 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:30.094265 8866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:30.094765 8866 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba92d7677fde4a65b2e8c9ad1c69da57" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39897 } }
I20260619 02:53:30.094864 8866 leader_election.cc:302] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba92d7677fde4a65b2e8c9ad1c69da57; no voters:
I20260619 02:53:30.094986 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 8798
I20260619 02:53:30.095031 8866 leader_election.cc:288] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:30.095115 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:30.095096 8869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:30.095229 8869 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [term 1 LEADER]: Becoming Leader. State: Replica: ba92d7677fde4a65b2e8c9ad1c69da57, State: Running, Role: LEADER
I20260619 02:53:30.095275 8866 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:30.095309 22163 external_mini_cluster.cc:1428] Setting key 7918550906dd1205b7522dde270eb776
I20260619 02:53:30.095316 8869 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba92d7677fde4a65b2e8c9ad1c69da57" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39897 } }
I20260619 02:53:30.095657 8869 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba92d7677fde4a65b2e8c9ad1c69da57. Latest consensus state: current_term: 1 leader_uuid: "ba92d7677fde4a65b2e8c9ad1c69da57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba92d7677fde4a65b2e8c9ad1c69da57" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39897 } } }
I20260619 02:53:30.095734 8869 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:30.096009 8870 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba92d7677fde4a65b2e8c9ad1c69da57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba92d7677fde4a65b2e8c9ad1c69da57" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 39897 } } }
I20260619 02:53:30.096071 8870 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:30.096464 8877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:30.097074 8877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:30.098906 8877 catalog_manager.cc:1347] Generated new cluster ID: e4e362426ff5449aa007005b8053790b
I20260619 02:53:30.098948 8877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:53:30 dist-test-slave-jt0t krb5kdc[8785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:30.106494 8877 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:30.107034 8877 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:30.107049 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.096066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.21.164.254:39897)
0619 02:53:30.096478 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.096487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.096519 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:53:30.096669 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.096678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.096745 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.096915 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:30.097614 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.098805 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.099871 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.100475 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.100578 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.103273 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:30.103279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:30.103279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:30.103294 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:30.105296 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:30.105698 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:30.105701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:30.105701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:30.105826 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:30.106042 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:30.106043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:30.106043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:30.106279 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:30.106452 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.106663 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.106814 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":74,"threads_started":1}
I20260619 02:53:30.109827 8877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57: Generated new TSK 0
I20260619 02:53:30.109990 8877 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.164.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
Entry for principal kudu/127.21.164.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab.
I20260619 02:53:30.136595 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39897
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:30.217487 8892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:30.217692 8892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:30.217715 8892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:30.218860 8892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:30.218904 8892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:30.218931 8892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:53:30.220399 8892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.193.keytab
--principal=kudu/127.21.164.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:30.220624 8892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:30.220861 8892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:30.226078 8900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:30.226200 8892 server_base.cc:1034] running on GCE node
W20260619 02:53:30.226116 8898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:30.226081 8897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:30.226580 8892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:30.226809 8892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:30.227957 8892 hybrid_clock.cc:648] HybridClock initialized: now 1781837610227940 us; error 29 us; skew 500 ppm
Jun 19 02:53:30 dist-test-slave-jt0t krb5kdc[8785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837610, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:30.230458 8892 init.cc:377] Logged in from keytab as kudu/127.21.164.193@KRBTEST.COM (short username kudu)
I20260619 02:53:30.230908 8892 webserver.cc:460] Webserver started at http://127.21.164.193:38077/ using document root <none> and password file <none>
I20260619 02:53:30.231127 8892 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:30.231171 8892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:30.231276 8892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:30.232389 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "1ff67d4662fe460298e00be14332c87d"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "a485bfc3b9523354d2c50e6c7303e6dc"
server_key_iv: "2dbb0759f4ab81f17e649e22bcfad0e2"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.232751 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "1ff67d4662fe460298e00be14332c87d"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "a485bfc3b9523354d2c50e6c7303e6dc"
server_key_iv: "2dbb0759f4ab81f17e649e22bcfad0e2"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.233999 8892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:30.234683 8907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.234858 8892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:30.234941 8892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "1ff67d4662fe460298e00be14332c87d"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "a485bfc3b9523354d2c50e6c7303e6dc"
server_key_iv: "2dbb0759f4ab81f17e649e22bcfad0e2"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.235010 8892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:30.257467 8892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:30.259747 8892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:30.259867 8892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:30.260075 8892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:30.260347 8892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:30.260376 8892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.260398 8892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:30.260412 8892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.265569 8892 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:40875
I20260619 02:53:30.265605 9020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:40875 every 8 connection(s)
I20260619 02:53:30.265913 8892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:30.271318 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 8892
I20260619 02:53:30.271440 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:30.271624 22163 external_mini_cluster.cc:1428] Setting key 8eaf95e99378197ef8ef24465929ccf6
Jun 19 02:53:30 dist-test-slave-jt0t krb5kdc[8785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837610, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.193@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:30.275207 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.266647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:42463 (local address 127.21.164.254:39897)
0619 02:53:30.266711 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.266713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.266869 (+ 156us) server_negotiation.cc:407] Connection header received
0619 02:53:30.267350 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.267352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.267364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.267423 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:30.268350 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.268874 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.269924 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.270141 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.270192 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.272428 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:30.272430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:30.272430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:30.272437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:30.273755 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:30.274228 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:30.274229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:30.274229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:30.274287 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:30.274584 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:30.274585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:30.274585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:30.274704 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:30.274763 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.275062 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.275138 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:53:30.275429 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.266601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39897 (local address 127.21.164.193:42463)
0619 02:53:30.266752 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.266794 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:30.267268 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:30.267487 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:30.267491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:30.268076 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:30.268270 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.268275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.268955 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.268956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.269841 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.269843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.270235 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.270236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.270346 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.270819 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:30.270827 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:30.272326 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:30.273831 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:30.273838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:30.273841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:30.274131 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:30.274354 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:30.274355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:30.274355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:30.274490 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:30.274766 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:30.274772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:30.274937 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 02:53:30.275049 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:53:30.275211 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260619 02:53:30.275938 9021 heartbeater.cc:347] Connected to a master server at 127.21.164.254:39897
I20260619 02:53:30.276043 9021 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:30.276245 9021 heartbeater.cc:511] Master 127.21.164.254:39897 requested a full tablet report, sending...
I20260619 02:53:30.276863 8830 ts_manager.cc:194] Registered new tserver with Master: 1ff67d4662fe460298e00be14332c87d (127.21.164.193:40875)
I20260619 02:53:30.277760 8830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.193@KRBTEST.COM'} at 127.21.164.193:42463
WARNING: no policy specified for kudu/127.21.164.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
Entry for principal kudu/127.21.164.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab.
I20260619 02:53:30.294943 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39897
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:30.375949 9026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:30.376173 9026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:30.376197 9026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:30.377501 9026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:30.377559 9026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:30.377588 9026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:53:30.379125 9026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.194.keytab
--principal=kudu/127.21.164.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:30.379387 9026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:30.379613 9026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:30.384582 9032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:30.384575 9034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:30.384742 9026 server_base.cc:1034] running on GCE node
W20260619 02:53:30.384575 9031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:30.385015 9026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:30.385277 9026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:30.386420 9026 hybrid_clock.cc:648] HybridClock initialized: now 1781837610386403 us; error 30 us; skew 500 ppm
Jun 19 02:53:30 dist-test-slave-jt0t krb5kdc[8785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837610, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:30.388880 9026 init.cc:377] Logged in from keytab as kudu/127.21.164.194@KRBTEST.COM (short username kudu)
I20260619 02:53:30.389323 9026 webserver.cc:460] Webserver started at http://127.21.164.194:36377/ using document root <none> and password file <none>
I20260619 02:53:30.389540 9026 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:30.389583 9026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:30.389714 9026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:30.390820 9026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "e44dd7ad0dea416daa8f8c5f2926d55b"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "de756676f9fc5e649f0d900aba167e3c"
server_key_iv: "b37ca158602dcdc6e18c442c096226ab"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.391156 9026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "e44dd7ad0dea416daa8f8c5f2926d55b"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "de756676f9fc5e649f0d900aba167e3c"
server_key_iv: "b37ca158602dcdc6e18c442c096226ab"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.392375 9026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:30.393060 9041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.393216 9026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.393301 9026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "e44dd7ad0dea416daa8f8c5f2926d55b"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "de756676f9fc5e649f0d900aba167e3c"
server_key_iv: "b37ca158602dcdc6e18c442c096226ab"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.393353 9026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:30.409864 9026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:30.411986 9026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:30.412101 9026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:30.412297 9026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:30.412551 9026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:30.412581 9026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.412601 9026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:30.412612 9026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.418583 9026 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:45065
I20260619 02:53:30.418663 9154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:45065 every 8 connection(s)
I20260619 02:53:30.418946 9026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:30.420258 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 9026
I20260619 02:53:30.420379 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:30.420593 22163 external_mini_cluster.cc:1428] Setting key f45f4c5cd3d6744eb527ba20903c5416
Jun 19 02:53:30 dist-test-slave-jt0t krb5kdc[8785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837610, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.194@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.164.195@KRBTEST.COM; defaulting to no policy
I20260619 02:53:30.428907 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.419912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:33191 (local address 127.21.164.254:39897)
0619 02:53:30.420005 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.420007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.420270 (+ 263us) server_negotiation.cc:407] Connection header received
0619 02:53:30.420916 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.420917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.420929 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.420983 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:30.422225 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.422773 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.423817 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.424008 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.424064 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.426354 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:30.426356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:30.426356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:30.426364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:30.427603 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:30.427956 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:30.427957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:30.427958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:30.428010 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:30.428262 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:30.428263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:30.428263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:30.428378 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:30.428434 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.428721 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.428813 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:53:30.429117 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.419847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39897 (local address 127.21.164.194:33191)
0619 02:53:30.420088 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.420145 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:30.420832 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:30.421061 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:30.421065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:30.421797 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:30.422100 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.422106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.422869 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.422871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.423718 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.423729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.424085 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.424086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.424214 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.424730 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:30.424738 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:30.426263 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:30.427676 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:30.427683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:30.427685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:30.427894 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:30.428099 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:30.428100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:30.428100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:30.428203 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:30.428434 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:30.428441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:30.428582 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 02:53:30.428694 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:53:30.428860 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260619 02:53:30.429589 9155 heartbeater.cc:347] Connected to a master server at 127.21.164.254:39897
I20260619 02:53:30.429673 9155 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:30.429844 9155 heartbeater.cc:511] Master 127.21.164.254:39897 requested a full tablet report, sending...
I20260619 02:53:30.430297 8830 ts_manager.cc:194] Registered new tserver with Master: e44dd7ad0dea416daa8f8c5f2926d55b (127.21.164.194:45065)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.195@KRBTEST.COM" created.
I20260619 02:53:30.430670 8830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.194@KRBTEST.COM'} at 127.21.164.194:33191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
Entry for principal kudu/127.21.164.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab.
I20260619 02:53:30.444833 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39897
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:30.526382 9160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:30.526590 9160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:30.526615 9160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:30.527714 9160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:30.527747 9160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:30.527772 9160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:53:30.529210 9160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.195.keytab
--principal=kudu/127.21.164.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:30.529398 9160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:30.529579 9160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:30.534405 9165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:30.534421 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:30.534422 9166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:30.534832 9160 server_base.cc:1034] running on GCE node
I20260619 02:53:30.535045 9160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:30.535276 9160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:30.536407 9160 hybrid_clock.cc:648] HybridClock initialized: now 1781837610536386 us; error 31 us; skew 500 ppm
Jun 19 02:53:30 dist-test-slave-jt0t krb5kdc[8785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837610, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:30.538936 9160 init.cc:377] Logged in from keytab as kudu/127.21.164.195@KRBTEST.COM (short username kudu)
I20260619 02:53:30.539405 9160 webserver.cc:460] Webserver started at http://127.21.164.195:35629/ using document root <none> and password file <none>
I20260619 02:53:30.539614 9160 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:30.539659 9160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:30.539778 9160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:30.540890 9160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "209450aeb07d4bfb81227e7afbae10b7"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "40946d71b772ef038e03c87f2d8afeb4"
server_key_iv: "7aab111b53156f82c7e2595880d9e82c"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.541213 9160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "209450aeb07d4bfb81227e7afbae10b7"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "40946d71b772ef038e03c87f2d8afeb4"
server_key_iv: "7aab111b53156f82c7e2595880d9e82c"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.542428 9160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:30.543077 9175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.543244 9160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:30.543327 9160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "209450aeb07d4bfb81227e7afbae10b7"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "40946d71b772ef038e03c87f2d8afeb4"
server_key_iv: "7aab111b53156f82c7e2595880d9e82c"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.543380 9160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:30.555034 9160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:30.557076 9160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:30.557180 9160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:30.557356 9160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:30.557636 9160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:30.557668 9160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.557690 9160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:30.557713 9160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.563066 9160 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:45537
I20260619 02:53:30.563107 9288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:45537 every 8 connection(s)
I20260619 02:53:30.563400 9160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
Jun 19 02:53:30 dist-test-slave-jt0t krb5kdc[8785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837610, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.195@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:30.569689 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 9160
I20260619 02:53:30.569808 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:30.570022 22163 external_mini_cluster.cc:1428] Setting key 6abe475b9d58c529a429e25507a0d49e
I20260619 02:53:30.572516 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.564130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:57285 (local address 127.21.164.254:39897)
0619 02:53:30.564180 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.564183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.564375 (+ 192us) server_negotiation.cc:407] Connection header received
0619 02:53:30.564910 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.564911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.564923 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.565015 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:30.565973 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.566457 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.567406 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.567594 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.567650 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.569698 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:30.569699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:30.569700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:30.569707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:30.571230 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:30.571620 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:30.571621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:30.571622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:30.571694 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:30.571979 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:30.571980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:30.571980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:30.572109 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:30.572169 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.572377 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.572451 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:53:30.572737 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.564078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:39897 (local address 127.21.164.195:57285)
0619 02:53:30.564256 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.564293 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:30.564811 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:30.565103 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:30.565107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:30.565679 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:30.565868 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.565873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.566537 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.566538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.567320 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.567322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.567660 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.567661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.567785 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.568243 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:30.568252 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:30.569598 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:30.571311 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:30.571316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:30.571318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:30.571542 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:30.571779 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:30.571780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:30.571781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:30.571892 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:30.572165 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:30.572168 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:30.572284 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 02:53:30.572363 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 02:53:30.572503 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260619 02:53:30.573171 9289 heartbeater.cc:347] Connected to a master server at 127.21.164.254:39897
I20260619 02:53:30.573261 9289 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:30.573436 9289 heartbeater.cc:511] Master 127.21.164.254:39897 requested a full tablet report, sending...
I20260619 02:53:30.573781 8830 ts_manager.cc:194] Registered new tserver with Master: 209450aeb07d4bfb81227e7afbae10b7 (127.21.164.195:45537)
I20260619 02:53:30.574168 8830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.164.195@KRBTEST.COM'} at 127.21.164.195:57285
I20260619 02:53:30.581772 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:30.588960 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.583185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.21.164.254:39897)
0619 02:53:30.583270 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.583272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.583342 (+ 70us) server_negotiation.cc:407] Connection header received
0619 02:53:30.583410 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.583411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.583422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.583471 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:30.584066 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.584549 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.585279 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.585467 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.585522 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.586648 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:30.586649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:30.586650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:30.586657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:30.587894 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:30.588197 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:30.588198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:30.588198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:30.588249 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:30.588407 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:30.588407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:30.588408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:30.588560 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:30.588642 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.588792 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.588870 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:53:30.590528 8830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:30.592067 8830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:30.598294 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.594239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:45065 (local address 127.0.0.1:40900)
0619 02:53:30.594400 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.594407 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:30.594442 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:30.594806 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:30.594807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:30.594809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:30.594855 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.594857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.596153 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.596154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.597112 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.597115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.598006 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.598007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.598081 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.598098 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 02:53:30.598162 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:53:30.598216 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":21,"thread_start_us":46,"threads_started":1}
I20260619 02:53:30.598403 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.594039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:45537 (local address 127.0.0.1:39822)
0619 02:53:30.594154 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.594191 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:30.594248 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:30.594662 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:30.594665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:30.594667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:30.594735 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.594736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.596118 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.596118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.597296 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.597298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.598165 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.598166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.598218 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.598222 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:30.598249 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 02:53:30.598279 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":30,"thread_start_us":69,"threads_started":1}
I20260619 02:53:30.598541 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.594170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.21.164.193:40875)
0619 02:53:30.594574 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.594579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.594590 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:53:30.594629 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.594631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.594646 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.594755 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:30.594875 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.595809 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.597351 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.598036 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.598126 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.598173 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.598270 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.598307 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":81,"threads_started":1}
I20260619 02:53:30.598567 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.594074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:40875 (local address 127.0.0.1:58422)
0619 02:53:30.594196 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.594206 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:30.594256 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:30.594758 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:30.594759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:30.594760 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:30.594796 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.594798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.595903 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.595904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.597259 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.597262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.598132 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.598133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.598199 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.598203 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:30.598255 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:53:30.598301 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":18,"thread_start_us":66,"threads_started":1}
I20260619 02:53:30.598603 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.594430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.21.164.194:45065)
0619 02:53:30.594603 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.594606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.594614 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:30.594646 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.594648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.594661 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.594726 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:30.594929 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.596022 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.597188 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.597937 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.598009 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.598079 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.598489 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.598531 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
I20260619 02:53:30.598837 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.594347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.21.164.195:45537)
0619 02:53:30.594523 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.594529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.594538 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:30.594572 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.594573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.594585 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.594646 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:30.594844 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.596026 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.597470 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.598073 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.598379 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.598690 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.598715 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.598758 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260619 02:53:30.599469 9223 tablet_service.cc:1467] Processing CreateTablet for tablet 420dc4c0e47a43fab91cf3caa9b36f89 (DEFAULT_TABLE table=test-table [id=a3ce9aee92a740a5b090f8de5bea4202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:30.599470 9089 tablet_service.cc:1467] Processing CreateTablet for tablet 420dc4c0e47a43fab91cf3caa9b36f89 (DEFAULT_TABLE table=test-table [id=a3ce9aee92a740a5b090f8de5bea4202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:30.599826 9223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420dc4c0e47a43fab91cf3caa9b36f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:30.599828 9089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420dc4c0e47a43fab91cf3caa9b36f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:30.601102 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 420dc4c0e47a43fab91cf3caa9b36f89 (DEFAULT_TABLE table=test-table [id=a3ce9aee92a740a5b090f8de5bea4202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:30.601392 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420dc4c0e47a43fab91cf3caa9b36f89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:30.603256 9303 tablet_bootstrap.cc:492] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b: Bootstrap starting.
I20260619 02:53:30.603778 9303 tablet_bootstrap.cc:654] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:30.603952 9305 tablet_bootstrap.cc:492] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d: Bootstrap starting.
I20260619 02:53:30.604347 9303 log.cc:824] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:30.604568 9305 tablet_bootstrap.cc:654] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:30.604741 9304 tablet_bootstrap.cc:492] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7: Bootstrap starting.
I20260619 02:53:30.605157 9305 log.cc:824] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:30.605351 9304 tablet_bootstrap.cc:654] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:30.605796 9304 log.cc:824] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:30.605904 9305 tablet_bootstrap.cc:492] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d: No bootstrap required, opened a new log
I20260619 02:53:30.605973 9305 ts_tablet_manager.cc:1397] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:30.606299 9303 tablet_bootstrap.cc:492] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b: No bootstrap required, opened a new log
I20260619 02:53:30.606365 9303 ts_tablet_manager.cc:1397] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 02:53:30.607868 9305 raft_consensus.cc:348] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209450aeb07d4bfb81227e7afbae10b7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45537 } } peers { permanent_uuid: "1ff67d4662fe460298e00be14332c87d" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40875 } } peers { permanent_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45065 } }
I20260619 02:53:30.608059 9305 raft_consensus.cc:374] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:30.608095 9305 raft_consensus.cc:729] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff67d4662fe460298e00be14332c87d, State: Initialized, Role: FOLLOWER
I20260619 02:53:30.608215 9305 consensus_queue.cc:260] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209450aeb07d4bfb81227e7afbae10b7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45537 } } peers { permanent_uuid: "1ff67d4662fe460298e00be14332c87d" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40875 } } peers { permanent_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45065 } }
I20260619 02:53:30.608227 9303 raft_consensus.cc:348] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209450aeb07d4bfb81227e7afbae10b7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45537 } } peers { permanent_uuid: "1ff67d4662fe460298e00be14332c87d" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40875 } } peers { permanent_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45065 } }
I20260619 02:53:30.608390 9303 raft_consensus.cc:374] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:30.608435 9303 raft_consensus.cc:729] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44dd7ad0dea416daa8f8c5f2926d55b, State: Initialized, Role: FOLLOWER
I20260619 02:53:30.608494 9305 ts_tablet_manager.cc:1428] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 02:53:30.608546 9021 heartbeater.cc:503] Master 127.21.164.254:39897 was elected leader, sending a full tablet report...
I20260619 02:53:30.608583 9303 consensus_queue.cc:260] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209450aeb07d4bfb81227e7afbae10b7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45537 } } peers { permanent_uuid: "1ff67d4662fe460298e00be14332c87d" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40875 } } peers { permanent_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45065 } }
I20260619 02:53:30.608868 9303 ts_tablet_manager.cc:1428] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 02:53:30.608894 9155 heartbeater.cc:503] Master 127.21.164.254:39897 was elected leader, sending a full tablet report...
I20260619 02:53:30.609736 9304 tablet_bootstrap.cc:492] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7: No bootstrap required, opened a new log
I20260619 02:53:30.609856 9304 ts_tablet_manager.cc:1397] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 02:53:30.611301 9304 raft_consensus.cc:348] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209450aeb07d4bfb81227e7afbae10b7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45537 } } peers { permanent_uuid: "1ff67d4662fe460298e00be14332c87d" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40875 } } peers { permanent_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45065 } }
I20260619 02:53:30.611435 9304 raft_consensus.cc:374] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:30.611464 9304 raft_consensus.cc:729] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209450aeb07d4bfb81227e7afbae10b7, State: Initialized, Role: FOLLOWER
I20260619 02:53:30.611593 9304 consensus_queue.cc:260] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209450aeb07d4bfb81227e7afbae10b7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45537 } } peers { permanent_uuid: "1ff67d4662fe460298e00be14332c87d" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40875 } } peers { permanent_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45065 } }
I20260619 02:53:30.611820 9304 ts_tablet_manager.cc:1428] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:53:30.611845 9289 heartbeater.cc:503] Master 127.21.164.254:39897 was elected leader, sending a full tablet report...
I20260619 02:53:30.640785 9309 raft_consensus.cc:482] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:30.640908 9309 raft_consensus.cc:504] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209450aeb07d4bfb81227e7afbae10b7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45537 } } peers { permanent_uuid: "1ff67d4662fe460298e00be14332c87d" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40875 } } peers { permanent_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45065 } }
I20260619 02:53:30.641273 9309 leader_election.cc:288] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 209450aeb07d4bfb81227e7afbae10b7 (127.21.164.195:45537), e44dd7ad0dea416daa8f8c5f2926d55b (127.21.164.194:45065)
I20260619 02:53:30.644289 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.641445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:45537 (local address 127.21.164.193:43363)
0619 02:53:30.641490 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.641503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:30.641546 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:30.641876 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:30.641878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:30.641881 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:30.641961 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.641963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.642687 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.642687 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.643498 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.643500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.644036 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.644037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.644131 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.644134 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:30.644175 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 02:53:30.644211 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":16}
I20260619 02:53:30.644349 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.641533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:43363 (local address 127.21.164.195:45537)
0619 02:53:30.641721 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.641723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.641732 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:30.641760 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.641761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.641770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.641832 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:30.642056 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.642615 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.643599 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.643965 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.644029 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.644038 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.644232 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.644281 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260619 02:53:30.644392 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.641645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:34101 (local address 127.21.164.194:45065)
0619 02:53:30.641711 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.641713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.641854 (+ 141us) server_negotiation.cc:407] Connection header received
0619 02:53:30.641898 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.641899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.641917 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.641995 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:30.642208 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.642672 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.643596 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.643961 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.644021 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.644031 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.644291 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.644334 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:53:30.644399 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.641569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:45065 (local address 127.21.164.193:34101)
0619 02:53:30.641771 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.641780 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:30.641833 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:30.642040 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:30.642041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:30.642043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:30.642136 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.642138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.642745 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.642745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.643500 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:30.643501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.644038 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.644039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:30.644199 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.644205 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:53:30.644272 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 02:53:30.644309 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260619 02:53:30.644737 9109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420dc4c0e47a43fab91cf3caa9b36f89" candidate_uuid: "1ff67d4662fe460298e00be14332c87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" is_pre_election: true
I20260619 02:53:30.644738 9243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420dc4c0e47a43fab91cf3caa9b36f89" candidate_uuid: "1ff67d4662fe460298e00be14332c87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209450aeb07d4bfb81227e7afbae10b7" is_pre_election: true
I20260619 02:53:30.644904 9109 raft_consensus.cc:2454] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ff67d4662fe460298e00be14332c87d in term 0.
I20260619 02:53:30.644903 9243 raft_consensus.cc:2454] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ff67d4662fe460298e00be14332c87d in term 0.
I20260619 02:53:30.645144 8910 leader_election.cc:302] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff67d4662fe460298e00be14332c87d, 209450aeb07d4bfb81227e7afbae10b7; no voters:
I20260619 02:53:30.645279 9309 raft_consensus.cc:2781] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:30.645344 9309 raft_consensus.cc:482] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:30.645361 9309 raft_consensus.cc:3037] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:30.646129 9309 raft_consensus.cc:504] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209450aeb07d4bfb81227e7afbae10b7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45537 } } peers { permanent_uuid: "1ff67d4662fe460298e00be14332c87d" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40875 } } peers { permanent_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45065 } }
I20260619 02:53:30.646265 9309 leader_election.cc:288] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [CANDIDATE]: Term 1 election: Requested vote from peers 209450aeb07d4bfb81227e7afbae10b7 (127.21.164.195:45537), e44dd7ad0dea416daa8f8c5f2926d55b (127.21.164.194:45065)
I20260619 02:53:30.646471 9243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420dc4c0e47a43fab91cf3caa9b36f89" candidate_uuid: "1ff67d4662fe460298e00be14332c87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209450aeb07d4bfb81227e7afbae10b7"
I20260619 02:53:30.646483 9109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420dc4c0e47a43fab91cf3caa9b36f89" candidate_uuid: "1ff67d4662fe460298e00be14332c87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b"
I20260619 02:53:30.646551 9243 raft_consensus.cc:3037] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:30.646553 9109 raft_consensus.cc:3037] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:30.647425 9109 raft_consensus.cc:2454] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff67d4662fe460298e00be14332c87d in term 1.
I20260619 02:53:30.647425 9243 raft_consensus.cc:2454] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff67d4662fe460298e00be14332c87d in term 1.
I20260619 02:53:30.647626 8910 leader_election.cc:302] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff67d4662fe460298e00be14332c87d, e44dd7ad0dea416daa8f8c5f2926d55b; no voters:
I20260619 02:53:30.647714 9309 raft_consensus.cc:2781] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:30.647934 9309 raft_consensus.cc:686] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 1 LEADER]: Becoming Leader. State: Replica: 1ff67d4662fe460298e00be14332c87d, State: Running, Role: LEADER
I20260619 02:53:30.648027 9309 consensus_queue.cc:237] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209450aeb07d4bfb81227e7afbae10b7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45537 } } peers { permanent_uuid: "1ff67d4662fe460298e00be14332c87d" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40875 } } peers { permanent_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45065 } }
I20260619 02:53:30.648898 8830 catalog_manager.cc:5495] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ff67d4662fe460298e00be14332c87d (127.21.164.193). New cstate: current_term: 1 leader_uuid: "1ff67d4662fe460298e00be14332c87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209450aeb07d4bfb81227e7afbae10b7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ff67d4662fe460298e00be14332c87d" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 40875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45065 } health_report { overall_health: UNKNOWN } } }
I20260619 02:53:30.655716 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.653588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.21.164.193:40875)
0619 02:53:30.653645 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.653647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.653655 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:30.653686 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.653687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.653704 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.653760 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:30.653846 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.654299 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.655065 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.655249 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.655293 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.655396 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:30.655492 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:30.655547 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.655625 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.655642 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:53:30.657442 9109 raft_consensus.cc:1261] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b [term 1 FOLLOWER]: Refusing update from remote peer 1ff67d4662fe460298e00be14332c87d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:30.657442 9243 raft_consensus.cc:1261] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7 [term 1 FOLLOWER]: Refusing update from remote peer 1ff67d4662fe460298e00be14332c87d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:30.657816 9313 consensus_queue.cc:1035] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [LEADER]: Connected to new peer: Peer: permanent_uuid: "209450aeb07d4bfb81227e7afbae10b7" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:30.657941 9309 consensus_queue.cc:1035] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e44dd7ad0dea416daa8f8c5f2926d55b" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 45065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:30.660197 9316 mvcc.cc:205] Tried to move back new op lower bound from 7298406853250359296 to 7298406853215002624. Current Snapshot: MvccSnapshot[applied={T|T < 7298406853250359296}]
I20260619 02:53:30.660209 9317 mvcc.cc:205] Tried to move back new op lower bound from 7298406853250359296 to 7298406853215002624. Current Snapshot: MvccSnapshot[applied={T|T < 7298406853250359296}]
I20260619 02:53:30.660874 9318 mvcc.cc:205] Tried to move back new op lower bound from 7298406853250359296 to 7298406853215002624. Current Snapshot: MvccSnapshot[applied={T|T < 7298406853250359296}]
I20260619 02:53:30.662411 8830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:30.662508 8830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:30.663210 8830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba92d7677fde4a65b2e8c9ad1c69da57: Sending DeleteTablet for 3 replicas of tablet 420dc4c0e47a43fab91cf3caa9b36f89
I20260619 02:53:30.663489 9223 tablet_service.cc:1514] Processing DeleteTablet for tablet 420dc4c0e47a43fab91cf3caa9b36f89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:30 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:39822
I20260619 02:53:30.663509 8955 tablet_service.cc:1514] Processing DeleteTablet for tablet 420dc4c0e47a43fab91cf3caa9b36f89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:30 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:58422
I20260619 02:53:30.663710 9089 tablet_service.cc:1514] Processing DeleteTablet for tablet 420dc4c0e47a43fab91cf3caa9b36f89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:30 UTC) from {username='kudu', principal='kudu/127.21.164.254@KRBTEST.COM'} at 127.0.0.1:40900
I20260619 02:53:30.663924 9326 tablet_replica.cc:330] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d: stopping tablet replica
I20260619 02:53:30.664024 9326 raft_consensus.cc:2229] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:53:30.664031 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 8892
I20260619 02:53:30.664137 9326 raft_consensus.cc:2258] T 420dc4c0e47a43fab91cf3caa9b36f89 P 1ff67d4662fe460298e00be14332c87d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:30.664559 9327 tablet_replica.cc:330] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b: stopping tablet replica
I20260619 02:53:30.664693 9327 raft_consensus.cc:2229] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:30.664793 9327 raft_consensus.cc:2258] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:30.664960 9325 tablet_replica.cc:330] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7: stopping tablet replica
I20260619 02:53:30.665058 9325 raft_consensus.cc:2229] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:30.665162 9325 raft_consensus.cc:2258] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:30.665377 9327 ts_tablet_manager.cc:1905] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:30.665575 9325 ts_tablet_manager.cc:1905] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:30.666896 9327 ts_tablet_manager.cc:1918] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:30.666963 9327 log.cc:1196] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-1/wal/wals/420dc4c0e47a43fab91cf3caa9b36f89
I20260619 02:53:30.667040 9325 ts_tablet_manager.cc:1918] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:30.667109 9325 log.cc:1196] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/420dc4c0e47a43fab91cf3caa9b36f89
I20260619 02:53:30.667251 9327 ts_tablet_manager.cc:1939] T 420dc4c0e47a43fab91cf3caa9b36f89 P e44dd7ad0dea416daa8f8c5f2926d55b: Deleting consensus metadata
I20260619 02:53:30.667356 9325 ts_tablet_manager.cc:1939] T 420dc4c0e47a43fab91cf3caa9b36f89 P 209450aeb07d4bfb81227e7afbae10b7: Deleting consensus metadata
I20260619 02:53:30.667637 8816 catalog_manager.cc:4841] TS e44dd7ad0dea416daa8f8c5f2926d55b (127.21.164.194:45065): tablet 420dc4c0e47a43fab91cf3caa9b36f89 (table test-table [id=a3ce9aee92a740a5b090f8de5bea4202]) successfully deleted
I20260619 02:53:30.668776 8816 catalog_manager.cc:4841] TS 209450aeb07d4bfb81227e7afbae10b7 (127.21.164.195:45537): tablet 420dc4c0e47a43fab91cf3caa9b36f89 (table test-table [id=a3ce9aee92a740a5b090f8de5bea4202]) successfully deleted
W20260619 02:53:30.670269 8814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:40875 (error 108)
I20260619 02:53:30.670542 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 9026
I20260619 02:53:30.670691 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.670564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:40875 (local address 127.0.0.1:58438)
0619 02:53:30.670620 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:30.670657 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:40875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 02:53:30.671268 8814 catalog_manager.cc:4584] TS 1ff67d4662fe460298e00be14332c87d (127.21.164.193:40875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 420dc4c0e47a43fab91cf3caa9b36f89: Network error: Client connection negotiation failed: client connection to 127.21.164.193:40875: connect: Connection refused (error 111)
I20260619 02:53:30.674928 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 9160
I20260619 02:53:30.678812 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 8798
2026-06-19T02:53:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1799 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T02:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:30Z Disabled control of system clock
I20260619 02:53:30.704103 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34719
--webserver_interface=127.21.164.254
--webserver_port=0
--builtin_ntp_servers=127.21.164.212:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:34719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:30.782518 9332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:30.782752 9332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:30.782773 9332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:30.783910 9332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:30.783947 9332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:30.783959 9332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:30.783970 9332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:30.783980 9332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:30.785380 9332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:34719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:34719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:30.785604 9332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:30.785861 9332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:30.790547 9337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:30.790555 9338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:30.790555 9340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:30.790632 9332 server_base.cc:1034] running on GCE node
I20260619 02:53:30.790947 9332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:30.791267 9332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:30.792407 9332 hybrid_clock.cc:648] HybridClock initialized: now 1781837610792393 us; error 30 us; skew 500 ppm
I20260619 02:53:30.793571 9332 webserver.cc:460] Webserver started at http://127.21.164.254:42453/ using document root <none> and password file <none>
I20260619 02:53:30.793797 9332 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:30.793846 9332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:30.793967 9332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:30.795017 9332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "a3f1d31d0c7546b4b45f13a813ba53aa"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "8b0d9d605d3ae8d705d9e19d13e89da3"
server_key_iv: "f7bffa719449c1286f6a74f21e0a523e"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.795356 9332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "a3f1d31d0c7546b4b45f13a813ba53aa"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "8b0d9d605d3ae8d705d9e19d13e89da3"
server_key_iv: "f7bffa719449c1286f6a74f21e0a523e"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.796550 9332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:30.797256 9346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.797407 9332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.797488 9332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "a3f1d31d0c7546b4b45f13a813ba53aa"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "8b0d9d605d3ae8d705d9e19d13e89da3"
server_key_iv: "f7bffa719449c1286f6a74f21e0a523e"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.797554 9332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:30.824931 9332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:30.825261 9332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:30.825393 9332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:30.829217 9332 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:34719
I20260619 02:53:30.829257 9398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:34719 every 8 connection(s)
I20260619 02:53:30.829574 9332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:30.830494 9399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:30.832700 9399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa: Bootstrap starting.
I20260619 02:53:30.833226 9399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:30.833451 9399 log.cc:824] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:30.834076 9399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa: No bootstrap required, opened a new log
I20260619 02:53:30.835597 9399 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f1d31d0c7546b4b45f13a813ba53aa" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34719 } }
I20260619 02:53:30.835736 9399 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:30.835757 9399 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f1d31d0c7546b4b45f13a813ba53aa, State: Initialized, Role: FOLLOWER
I20260619 02:53:30.835872 9399 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f1d31d0c7546b4b45f13a813ba53aa" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34719 } }
I20260619 02:53:30.835947 9399 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:30.835973 9399 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:30.836015 9399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:30.836504 9399 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f1d31d0c7546b4b45f13a813ba53aa" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34719 } }
I20260619 02:53:30.836585 9399 leader_election.cc:302] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3f1d31d0c7546b4b45f13a813ba53aa; no voters:
I20260619 02:53:30.836752 9399 leader_election.cc:288] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:30.836817 9402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:30.836966 9402 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [term 1 LEADER]: Becoming Leader. State: Replica: a3f1d31d0c7546b4b45f13a813ba53aa, State: Running, Role: LEADER
I20260619 02:53:30.836999 9399 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:30.837057 9402 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f1d31d0c7546b4b45f13a813ba53aa" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34719 } }
I20260619 02:53:30.837519 9403 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3f1d31d0c7546b4b45f13a813ba53aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f1d31d0c7546b4b45f13a813ba53aa" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34719 } } }
I20260619 02:53:30.837605 9403 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:30.837754 9404 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3f1d31d0c7546b4b45f13a813ba53aa. Latest consensus state: current_term: 1 leader_uuid: "a3f1d31d0c7546b4b45f13a813ba53aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f1d31d0c7546b4b45f13a813ba53aa" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 34719 } } }
I20260619 02:53:30.837803 9411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:30.837813 9404 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:30.838364 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 9332
I20260619 02:53:30.838380 9411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:30.838472 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:30.838672 22163 external_mini_cluster.cc:1428] Setting key a127b74a7710c2fd2ff3cbb739c2b789
I20260619 02:53:30.840088 9411 catalog_manager.cc:1347] Generated new cluster ID: 7bd132aea48440f780049d4ebf2ecf07
I20260619 02:53:30.840144 9411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:30.844949 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:30.839116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.21.164.254:34719)
0619 02:53:30.839306 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:30.839315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:30.839606 (+ 291us) server_negotiation.cc:407] Connection header received
0619 02:53:30.839742 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:30.839748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:30.839802 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:30.839922 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:30.840149 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.841531 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.842607 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:30.843217 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:30.843640 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:30.844068 (+ 428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:30.844072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:30.844072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:30.844278 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:30.844320 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:30.844333 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:30.844604 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:30.844636 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 02:53:30.844744 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260619 02:53:30.856416 9411 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:30.856985 9411 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:30.864221 9411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa: Generated new TSK 0
I20260619 02:53:30.864388 9411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:30.869655 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.193:0
--local_ip_for_outbound_sockets=127.21.164.193
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34719
--builtin_ntp_servers=127.21.164.212:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:30.973129 9423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:30.973347 9423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:30.973368 9423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:30.974828 9423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:30.974882 9423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:30.974910 9423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.193
I20260619 02:53:30.976495 9423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.164.193
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.193
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:30.976790 9423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:30.977098 9423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:30.982050 9429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:30.982080 9431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:30.982154 9428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:30.982606 9423 server_base.cc:1034] running on GCE node
I20260619 02:53:30.982824 9423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:30.983074 9423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:30.984228 9423 hybrid_clock.cc:648] HybridClock initialized: now 1781837610984217 us; error 27 us; skew 500 ppm
I20260619 02:53:30.985461 9423 webserver.cc:460] Webserver started at http://127.21.164.193:32967/ using document root <none> and password file <none>
I20260619 02:53:30.985671 9423 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:30.985713 9423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:30.985806 9423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:30.986869 9423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/data/instance:
uuid: "a11a1931381c409b950f7a50e8c7c490"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "b4685ecdbea1024c87557b66448f351c"
server_key_iv: "a4e538ace79f48aa401e25a525928b21"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.987208 9423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance:
uuid: "a11a1931381c409b950f7a50e8c7c490"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "b4685ecdbea1024c87557b66448f351c"
server_key_iv: "a4e538ace79f48aa401e25a525928b21"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.988432 9423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:53:30.989166 9437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.989292 9423 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:30.989368 9423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/wal
uuid: "a11a1931381c409b950f7a50e8c7c490"
format_stamp: "Formatted at 2026-06-19 02:53:30 on dist-test-slave-jt0t"
server_key: "b4685ecdbea1024c87557b66448f351c"
server_key_iv: "a4e538ace79f48aa401e25a525928b21"
server_key_version: "encryptionkey@0"
I20260619 02:53:30.989446 9423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:31.008570 9423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:31.008971 9423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:31.009101 9423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:31.009330 9423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:31.009658 9423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:31.009691 9423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:31.009716 9423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:31.009740 9423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:31.015199 9423 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.193:33629
I20260619 02:53:31.015297 9550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.193:33629 every 8 connection(s)
I20260619 02:53:31.015600 9423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/data/info.pb
I20260619 02:53:31.020830 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.016377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.193:37751 (local address 127.21.164.254:34719)
0619 02:53:31.016456 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:31.016457 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:31.016764 (+ 307us) server_negotiation.cc:407] Connection header received
0619 02:53:31.017314 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:31.017315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:31.017331 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:31.017383 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:31.018054 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.018568 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.019528 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.019725 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.019780 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.020550 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:31.020551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:31.020552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:31.020557 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:31.020572 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:31.020576 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:31.020620 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:31.020718 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 02:53:31.020749 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:53:31.021049 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.016358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34719 (local address 127.21.164.193:37751)
0619 02:53:31.016624 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.016678 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:31.017217 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:31.017453 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:31.017457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:31.017468 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:31.017970 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.017974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.018651 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.018652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.019447 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.019449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.019794 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.019795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.019927 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.020423 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:31.020424 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:31.020454 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:31.020477 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:31.020626 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:31.020631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:31.020634 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:31.020723 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 02:53:31.020827 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
I20260619 02:53:31.021545 9551 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34719
I20260619 02:53:31.021641 9551 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:31.021821 9551 heartbeater.cc:511] Master 127.21.164.254:34719 requested a full tablet report, sending...
I20260619 02:53:31.022382 9363 ts_manager.cc:194] Registered new tserver with Master: a11a1931381c409b950f7a50e8c7c490 (127.21.164.193:33629)
I20260619 02:53:31.023023 9363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.193:37751
I20260619 02:53:31.025346 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 9423
I20260619 02:53:31.025444 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-0/wal/instance
I20260619 02:53:31.025622 22163 external_mini_cluster.cc:1428] Setting key 9e4274e7948b2866ad7f514c6ea51f36
I20260619 02:53:31.026554 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.194:0
--local_ip_for_outbound_sockets=127.21.164.194
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34719
--builtin_ntp_servers=127.21.164.212:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 02:53:31.107566 9554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:31.107800 9554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:31.107828 9554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:31.109014 9554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:31.109052 9554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:31.109086 9554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.194
I20260619 02:53:31.110445 9554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.164.194
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.194
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:31.110692 9554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:31.110986 9554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:31.116003 9560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:31.116052 9559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:31.116178 9554 server_base.cc:1034] running on GCE node
W20260619 02:53:31.116284 9562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:31.116546 9554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:31.116855 9554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:31.118009 9554 hybrid_clock.cc:648] HybridClock initialized: now 1781837611117982 us; error 42 us; skew 500 ppm
I20260619 02:53:31.119287 9554 webserver.cc:460] Webserver started at http://127.21.164.194:38383/ using document root <none> and password file <none>
I20260619 02:53:31.119522 9554 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:31.119572 9554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:31.119696 9554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:31.120867 9554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/data/instance:
uuid: "436e34042a9d4adc961adf44c746ad97"
format_stamp: "Formatted at 2026-06-19 02:53:31 on dist-test-slave-jt0t"
server_key: "93592c6c82ce8d8fdc27ccd8e9c50862"
server_key_iv: "ad637e310820936cd36175f863754a57"
server_key_version: "encryptionkey@0"
I20260619 02:53:31.121246 9554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance:
uuid: "436e34042a9d4adc961adf44c746ad97"
format_stamp: "Formatted at 2026-06-19 02:53:31 on dist-test-slave-jt0t"
server_key: "93592c6c82ce8d8fdc27ccd8e9c50862"
server_key_iv: "ad637e310820936cd36175f863754a57"
server_key_version: "encryptionkey@0"
I20260619 02:53:31.122509 9554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:31.123217 9568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:31.123358 9554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:31.123438 9554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/wal
uuid: "436e34042a9d4adc961adf44c746ad97"
format_stamp: "Formatted at 2026-06-19 02:53:31 on dist-test-slave-jt0t"
server_key: "93592c6c82ce8d8fdc27ccd8e9c50862"
server_key_iv: "ad637e310820936cd36175f863754a57"
server_key_version: "encryptionkey@0"
I20260619 02:53:31.123504 9554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:31.131011 9554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:31.131281 9554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:31.131398 9554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:31.131603 9554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:31.131916 9554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:31.131950 9554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:31.131978 9554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:31.131995 9554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:31.138268 9554 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.194:44629
I20260619 02:53:31.138330 9681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.194:44629 every 8 connection(s)
I20260619 02:53:31.138671 9554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/data/info.pb
I20260619 02:53:31.141215 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 9554
I20260619 02:53:31.141337 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-1/wal/instance
I20260619 02:53:31.141525 22163 external_mini_cluster.cc:1428] Setting key b9730646a8e4a7a5f60de6f2c3ef2248
I20260619 02:53:31.142473 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.164.195:0
--local_ip_for_outbound_sockets=127.21.164.195
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34719
--builtin_ntp_servers=127.21.164.212:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 02:53:31.144846 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.139389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.194:50805 (local address 127.21.164.254:34719)
0619 02:53:31.139485 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:31.139487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:31.139672 (+ 185us) server_negotiation.cc:407] Connection header received
0619 02:53:31.140240 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:31.140241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:31.140252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:31.140313 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:31.141157 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.141763 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.143104 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.143318 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.143380 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.144530 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:31.144532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:31.144532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:31.144538 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:31.144557 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:31.144561 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:31.144615 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:31.144727 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 02:53:31.144762 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 02:53:31.145020 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.139375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34719 (local address 127.21.164.194:50805)
0619 02:53:31.139556 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.139593 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:31.140065 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:31.140427 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:31.140431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:31.140440 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:31.141024 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.141031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.141837 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.141839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.143004 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.143008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.143397 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.143398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.143551 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.144367 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:31.144367 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:31.144405 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:31.144431 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:31.144620 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:31.144623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:31.144625 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 02:53:31.144716 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 02:53:31.144813 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260619 02:53:31.145530 9682 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34719
I20260619 02:53:31.145637 9682 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:31.145946 9682 heartbeater.cc:511] Master 127.21.164.254:34719 requested a full tablet report, sending...
I20260619 02:53:31.146440 9363 ts_manager.cc:194] Registered new tserver with Master: 436e34042a9d4adc961adf44c746ad97 (127.21.164.194:44629)
I20260619 02:53:31.146804 9363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.194:50805
W20260619 02:53:31.223470 9685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:31.223673 9685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:31.223694 9685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:31.224870 9685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:31.224911 9685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:31.224936 9685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.164.195
I20260619 02:53:31.226432 9685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.164.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.164.195
--webserver_port=0
--tserver_master_addrs=127.21.164.254:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.164.195
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:31.226673 9685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:31.226954 9685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:31.231918 9691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:31.231925 9690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:31.231956 9685 server_base.cc:1034] running on GCE node
W20260619 02:53:31.231931 9693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:31.232352 9685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:31.232702 9685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:31.233888 9685 hybrid_clock.cc:648] HybridClock initialized: now 1781837611233871 us; error 29 us; skew 500 ppm
I20260619 02:53:31.235222 9685 webserver.cc:460] Webserver started at http://127.21.164.195:44147/ using document root <none> and password file <none>
I20260619 02:53:31.235462 9685 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:31.235512 9685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:31.235641 9685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:31.236792 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/data/instance:
uuid: "4021bb31d81b4e0db67a54ca39383a7e"
format_stamp: "Formatted at 2026-06-19 02:53:31 on dist-test-slave-jt0t"
server_key: "f4267d8f0d865a5f450d3dc4831dad55"
server_key_iv: "6cd1f3894e8fe2b4477a280f117bd0b1"
server_key_version: "encryptionkey@0"
I20260619 02:53:31.237126 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance:
uuid: "4021bb31d81b4e0db67a54ca39383a7e"
format_stamp: "Formatted at 2026-06-19 02:53:31 on dist-test-slave-jt0t"
server_key: "f4267d8f0d865a5f450d3dc4831dad55"
server_key_iv: "6cd1f3894e8fe2b4477a280f117bd0b1"
server_key_version: "encryptionkey@0"
I20260619 02:53:31.238451 9685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:31.239282 9699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:31.239468 9685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:31.239550 9685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/wal
uuid: "4021bb31d81b4e0db67a54ca39383a7e"
format_stamp: "Formatted at 2026-06-19 02:53:31 on dist-test-slave-jt0t"
server_key: "f4267d8f0d865a5f450d3dc4831dad55"
server_key_iv: "6cd1f3894e8fe2b4477a280f117bd0b1"
server_key_version: "encryptionkey@0"
I20260619 02:53:31.239624 9685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:31.253212 9685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:31.253551 9685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:31.253684 9685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:31.253909 9685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:53:31.254274 9685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:53:31.254312 9685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:31.254343 9685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:53:31.254360 9685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:31.260130 9685 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.195:34791
I20260619 02:53:31.260337 9812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.195:34791 every 8 connection(s)
I20260619 02:53:31.260550 9685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/data/info.pb
I20260619 02:53:31.265894 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.261350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:42303 (local address 127.21.164.254:34719)
0619 02:53:31.261437 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:31.261439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:31.261638 (+ 199us) server_negotiation.cc:407] Connection header received
0619 02:53:31.262134 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:31.262135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:31.262147 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:31.262211 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:31.262893 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.263490 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.264526 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.264785 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.264842 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.265658 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:31.265659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:31.265659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:31.265664 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:31.265681 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:31.265685 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:31.265739 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:31.265799 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 02:53:31.265828 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:53:31.266074 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.261331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:34719 (local address 127.21.164.195:42303)
0619 02:53:31.261506 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.261550 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:31.262041 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:31.262271 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:31.262274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:31.262283 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:31.262790 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.262794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.263586 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.263587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.264431 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.264434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.264849 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.264850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.264966 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.265505 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:53:31.265505 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:31.265536 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:53:31.265562 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:31.265742 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:31.265744 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:31.265747 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:31.265794 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 02:53:31.265881 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260619 02:53:31.266585 9813 heartbeater.cc:347] Connected to a master server at 127.21.164.254:34719
I20260619 02:53:31.266683 9813 heartbeater.cc:464] Registering TS with master...
I20260619 02:53:31.266846 9813 heartbeater.cc:511] Master 127.21.164.254:34719 requested a full tablet report, sending...
I20260619 02:53:31.267401 9363 ts_manager.cc:194] Registered new tserver with Master: 4021bb31d81b4e0db67a54ca39383a7e (127.21.164.195:34791)
I20260619 02:53:31.267891 9363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.164.195:42303
I20260619 02:53:31.268329 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 9685
I20260619 02:53:31.268438 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/wal/instance
I20260619 02:53:31.268674 22163 external_mini_cluster.cc:1428] Setting key de0c57a527ac70756f2717eea937877f
I20260619 02:53:31.269750 22163 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:53:31.273886 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.271101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.21.164.254:34719)
0619 02:53:31.271172 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:31.271175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:31.271257 (+ 82us) server_negotiation.cc:407] Connection header received
0619 02:53:31.271314 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:31.271314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:31.271327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:31.271373 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:31.271518 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.272018 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.272768 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.272963 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.273024 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.273643 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:31.273644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:31.273644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:31.273649 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:31.273664 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:31.273668 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:31.273713 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:31.273770 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 02:53:31.273798 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:53:31.275511 9363 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:53:31.277045 9363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:53:31.283205 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.279552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.21.164.195:34791)
0619 02:53:31.279720 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:31.279723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:31.279734 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:53:31.279768 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:31.279770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:31.279783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:31.279854 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:31.279980 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.281098 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.282343 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.282822 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.282889 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.282993 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:31.283072 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 02:53:31.283114 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260619 02:53:31.283234 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.279138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:44629 (local address 127.0.0.1:51696)
0619 02:53:31.279268 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.279313 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:31.279378 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:31.279805 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:31.279806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:31.279809 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:31.279857 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.279859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.280869 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.280870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.282095 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.282098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.282953 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.282955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.283015 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.283019 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:53:31.283048 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 02:53:31.283140 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260619 02:53:31.283232 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.279347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.195:34791 (local address 127.0.0.1:35554)
0619 02:53:31.279515 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.279524 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:31.279592 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:31.279864 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:31.279865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:31.279866 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:31.279898 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.279899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.281181 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.281182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.282277 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.282278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.282899 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.282900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.282974 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.282988 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 02:53:31.283059 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 02:53:31.283152 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":91,"thread_start_us":78,"threads_started":1}
I20260619 02:53:31.283432 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.279248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.21.164.193:33629)
0619 02:53:31.279596 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:31.279601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:31.279611 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:53:31.279651 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:31.279653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:31.279668 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:31.279741 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:31.279908 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.281099 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.282113 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.282860 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.282929 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.283054 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:31.283313 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 02:53:31.283356 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":99,"threads_started":1}
I20260619 02:53:31.283514 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.279331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.21.164.194:44629)
0619 02:53:31.279639 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:31.279643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:31.279653 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:53:31.279691 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:31.279693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:31.279709 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:31.279923 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:31.279988 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.280800 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.282169 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.282860 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.283277 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.283367 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:31.283390 (+ 23us) server_negotiation.cc:299] Negotiation successful
0619 02:53:31.283428 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":88,"threads_started":1}
I20260619 02:53:31.283599 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.279138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:33629 (local address 127.0.0.1:40238)
0619 02:53:31.279465 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.279488 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:31.279529 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:31.279756 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:31.279759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:31.279761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:31.279828 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.279830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.281190 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.281191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.282031 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.282033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.283085 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.283086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.283170 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.283173 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:31.283219 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 02:53:31.283252 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":36,"thread_start_us":77,"threads_started":1}
I20260619 02:53:31.284230 9616 tablet_service.cc:1467] Processing CreateTablet for tablet 870afcbb7c014bb9b178a96ab10a5943 (DEFAULT_TABLE table=test-table [id=13dd85424df84a89b4aa4844f7417309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:31.284229 9485 tablet_service.cc:1467] Processing CreateTablet for tablet 870afcbb7c014bb9b178a96ab10a5943 (DEFAULT_TABLE table=test-table [id=13dd85424df84a89b4aa4844f7417309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:31.284595 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870afcbb7c014bb9b178a96ab10a5943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:31.284595 9616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870afcbb7c014bb9b178a96ab10a5943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:31.285168 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 870afcbb7c014bb9b178a96ab10a5943 (DEFAULT_TABLE table=test-table [id=13dd85424df84a89b4aa4844f7417309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:53:31.285450 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870afcbb7c014bb9b178a96ab10a5943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:31.287945 9827 tablet_bootstrap.cc:492] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97: Bootstrap starting.
I20260619 02:53:31.288148 9828 tablet_bootstrap.cc:492] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490: Bootstrap starting.
I20260619 02:53:31.288285 9829 tablet_bootstrap.cc:492] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: Bootstrap starting.
I20260619 02:53:31.288852 9829 tablet_bootstrap.cc:654] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:31.288852 9828 tablet_bootstrap.cc:654] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:31.289178 9827 tablet_bootstrap.cc:654] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:31.289176 9828 log.cc:824] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:31.289175 9829 log.cc:824] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:31.289458 9827 log.cc:824] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:31.289924 9828 tablet_bootstrap.cc:492] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490: No bootstrap required, opened a new log
I20260619 02:53:31.289994 9828 ts_tablet_manager.cc:1397] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:31.290091 9829 tablet_bootstrap.cc:492] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: No bootstrap required, opened a new log
I20260619 02:53:31.290158 9829 ts_tablet_manager.cc:1397] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:31.290177 9827 tablet_bootstrap.cc:492] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97: No bootstrap required, opened a new log
I20260619 02:53:31.290225 9827 ts_tablet_manager.cc:1397] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:53:31.291944 9828 raft_consensus.cc:348] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11a1931381c409b950f7a50e8c7c490" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 33629 } } peers { permanent_uuid: "436e34042a9d4adc961adf44c746ad97" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 44629 } } peers { permanent_uuid: "4021bb31d81b4e0db67a54ca39383a7e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 34791 } }
I20260619 02:53:31.292127 9828 raft_consensus.cc:374] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:31.292162 9828 raft_consensus.cc:729] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11a1931381c409b950f7a50e8c7c490, State: Initialized, Role: FOLLOWER
I20260619 02:53:31.292299 9828 consensus_queue.cc:260] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11a1931381c409b950f7a50e8c7c490" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 33629 } } peers { permanent_uuid: "436e34042a9d4adc961adf44c746ad97" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 44629 } } peers { permanent_uuid: "4021bb31d81b4e0db67a54ca39383a7e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 34791 } }
I20260619 02:53:31.292546 9828 ts_tablet_manager.cc:1428] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:31.292688 9551 heartbeater.cc:503] Master 127.21.164.254:34719 was elected leader, sending a full tablet report...
I20260619 02:53:31.292703 9827 raft_consensus.cc:348] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11a1931381c409b950f7a50e8c7c490" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 33629 } } peers { permanent_uuid: "436e34042a9d4adc961adf44c746ad97" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 44629 } } peers { permanent_uuid: "4021bb31d81b4e0db67a54ca39383a7e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 34791 } }
I20260619 02:53:31.292697 9829 raft_consensus.cc:348] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11a1931381c409b950f7a50e8c7c490" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 33629 } } peers { permanent_uuid: "436e34042a9d4adc961adf44c746ad97" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 44629 } } peers { permanent_uuid: "4021bb31d81b4e0db67a54ca39383a7e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 34791 } }
I20260619 02:53:31.292857 9829 raft_consensus.cc:374] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:31.292857 9827 raft_consensus.cc:374] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:31.292897 9827 raft_consensus.cc:729] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436e34042a9d4adc961adf44c746ad97, State: Initialized, Role: FOLLOWER
I20260619 02:53:31.292897 9829 raft_consensus.cc:729] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4021bb31d81b4e0db67a54ca39383a7e, State: Initialized, Role: FOLLOWER
I20260619 02:53:31.293004 9829 consensus_queue.cc:260] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11a1931381c409b950f7a50e8c7c490" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 33629 } } peers { permanent_uuid: "436e34042a9d4adc961adf44c746ad97" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 44629 } } peers { permanent_uuid: "4021bb31d81b4e0db67a54ca39383a7e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 34791 } }
I20260619 02:53:31.293008 9827 consensus_queue.cc:260] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11a1931381c409b950f7a50e8c7c490" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 33629 } } peers { permanent_uuid: "436e34042a9d4adc961adf44c746ad97" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 44629 } } peers { permanent_uuid: "4021bb31d81b4e0db67a54ca39383a7e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 34791 } }
I20260619 02:53:31.293267 9829 ts_tablet_manager.cc:1428] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:31.293263 9827 ts_tablet_manager.cc:1428] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:53:31.293288 9682 heartbeater.cc:503] Master 127.21.164.254:34719 was elected leader, sending a full tablet report...
I20260619 02:53:31.293368 9813 heartbeater.cc:503] Master 127.21.164.254:34719 was elected leader, sending a full tablet report...
W20260619 02:53:31.389256 9683 tablet.cc:2369] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:31.469923 9835 raft_consensus.cc:482] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:53:31.470053 9835 raft_consensus.cc:504] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11a1931381c409b950f7a50e8c7c490" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 33629 } } peers { permanent_uuid: "436e34042a9d4adc961adf44c746ad97" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 44629 } } peers { permanent_uuid: "4021bb31d81b4e0db67a54ca39383a7e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 34791 } }
I20260619 02:53:31.470422 9835 leader_election.cc:288] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a11a1931381c409b950f7a50e8c7c490 (127.21.164.193:33629), 436e34042a9d4adc961adf44c746ad97 (127.21.164.194:44629)
I20260619 02:53:31.473130 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.470693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:47177 (local address 127.21.164.193:33629)
0619 02:53:31.470734 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:31.470736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:31.470745 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:31.470776 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:31.470777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:31.470788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:31.470834 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:31.470994 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.471501 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.472429 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.472818 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.472880 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.472889 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:31.473016 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 02:53:31.473055 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 02:53:31.473130 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.470597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:33629 (local address 127.21.164.195:47177)
0619 02:53:31.470667 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.470679 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:31.470723 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:31.470848 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:31.470849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:31.470850 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:31.470924 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.470926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.471588 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.471589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.472336 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.472338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.472890 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.472892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.472954 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.472957 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:31.473002 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 02:53:31.473055 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260619 02:53:31.473202 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.470688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.164.195:53125 (local address 127.21.164.194:44629)
0619 02:53:31.470733 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:31.470735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:31.470836 (+ 101us) server_negotiation.cc:407] Connection header received
0619 02:53:31.470889 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:31.470890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:31.470900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:31.470945 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:53:31.471120 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.471607 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.472517 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.472883 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.472943 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.472951 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:31.473095 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 02:53:31.473132 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:53:31.473254 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.470612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:44629 (local address 127.21.164.195:53125)
0619 02:53:31.470767 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.470775 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:31.470832 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:31.470986 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:31.470987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:31.470989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:53:31.471055 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.471057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.471685 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.471686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.472433 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:31.472434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.472965 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.472965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:31.473038 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.473041 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 02:53:31.473085 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 02:53:31.473123 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260619 02:53:31.473484 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870afcbb7c014bb9b178a96ab10a5943" candidate_uuid: "4021bb31d81b4e0db67a54ca39383a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11a1931381c409b950f7a50e8c7c490" is_pre_election: true
I20260619 02:53:31.473618 9505 raft_consensus.cc:2454] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4021bb31d81b4e0db67a54ca39383a7e in term 0.
I20260619 02:53:31.473870 9701 leader_election.cc:302] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4021bb31d81b4e0db67a54ca39383a7e, a11a1931381c409b950f7a50e8c7c490; no voters:
I20260619 02:53:31.473888 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870afcbb7c014bb9b178a96ab10a5943" candidate_uuid: "4021bb31d81b4e0db67a54ca39383a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436e34042a9d4adc961adf44c746ad97" is_pre_election: true
I20260619 02:53:31.474035 9636 raft_consensus.cc:2454] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4021bb31d81b4e0db67a54ca39383a7e in term 0.
I20260619 02:53:31.474090 9835 raft_consensus.cc:2781] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:53:31.474164 9835 raft_consensus.cc:482] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:53:31.474190 9835 raft_consensus.cc:3037] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:31.474876 9835 raft_consensus.cc:504] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11a1931381c409b950f7a50e8c7c490" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 33629 } } peers { permanent_uuid: "436e34042a9d4adc961adf44c746ad97" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 44629 } } peers { permanent_uuid: "4021bb31d81b4e0db67a54ca39383a7e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 34791 } }
I20260619 02:53:31.475010 9835 leader_election.cc:288] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [CANDIDATE]: Term 1 election: Requested vote from peers a11a1931381c409b950f7a50e8c7c490 (127.21.164.193:33629), 436e34042a9d4adc961adf44c746ad97 (127.21.164.194:44629)
I20260619 02:53:31.475152 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870afcbb7c014bb9b178a96ab10a5943" candidate_uuid: "4021bb31d81b4e0db67a54ca39383a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11a1931381c409b950f7a50e8c7c490"
I20260619 02:53:31.475200 9636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870afcbb7c014bb9b178a96ab10a5943" candidate_uuid: "4021bb31d81b4e0db67a54ca39383a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436e34042a9d4adc961adf44c746ad97"
I20260619 02:53:31.475220 9505 raft_consensus.cc:3037] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:31.475289 9636 raft_consensus.cc:3037] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:31.475956 9505 raft_consensus.cc:2454] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4021bb31d81b4e0db67a54ca39383a7e in term 1.
I20260619 02:53:31.476019 9636 raft_consensus.cc:2454] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4021bb31d81b4e0db67a54ca39383a7e in term 1.
I20260619 02:53:31.476105 9701 leader_election.cc:302] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4021bb31d81b4e0db67a54ca39383a7e, a11a1931381c409b950f7a50e8c7c490; no voters:
I20260619 02:53:31.476198 9835 raft_consensus.cc:2781] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:31.476354 9835 raft_consensus.cc:686] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 1 LEADER]: Becoming Leader. State: Replica: 4021bb31d81b4e0db67a54ca39383a7e, State: Running, Role: LEADER
I20260619 02:53:31.476459 9835 consensus_queue.cc:237] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11a1931381c409b950f7a50e8c7c490" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 33629 } } peers { permanent_uuid: "436e34042a9d4adc961adf44c746ad97" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 44629 } } peers { permanent_uuid: "4021bb31d81b4e0db67a54ca39383a7e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 34791 } }
I20260619 02:53:31.477339 9361 catalog_manager.cc:5495] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4021bb31d81b4e0db67a54ca39383a7e (127.21.164.195). New cstate: current_term: 1 leader_uuid: "4021bb31d81b4e0db67a54ca39383a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11a1931381c409b950f7a50e8c7c490" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "436e34042a9d4adc961adf44c746ad97" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4021bb31d81b4e0db67a54ca39383a7e" member_type: VOTER last_known_addr { host: "127.21.164.195" port: 34791 } health_report { overall_health: HEALTHY } } }
I20260619 02:53:31.509166 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.506875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.21.164.195:34791)
0619 02:53:31.506934 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:31.506936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:31.506944 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:31.506984 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:31.506985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:31.507007 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:31.507065 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:31.507196 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.507665 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.508394 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:31.508600 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:31.508683 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:31.508825 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:31.508934 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:31.509008 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:31.509039 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 02:53:31.509060 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:53:31.510993 9505 raft_consensus.cc:1261] T 870afcbb7c014bb9b178a96ab10a5943 P a11a1931381c409b950f7a50e8c7c490 [term 1 FOLLOWER]: Refusing update from remote peer 4021bb31d81b4e0db67a54ca39383a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:31.510987 9636 raft_consensus.cc:1261] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97 [term 1 FOLLOWER]: Refusing update from remote peer 4021bb31d81b4e0db67a54ca39383a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 02:53:31.511080 9814 tablet.cc:2369] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:31.511360 9835 consensus_queue.cc:1035] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "436e34042a9d4adc961adf44c746ad97" member_type: VOTER last_known_addr { host: "127.21.164.194" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:31.511511 9837 consensus_queue.cc:1035] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a11a1931381c409b950f7a50e8c7c490" member_type: VOTER last_known_addr { host: "127.21.164.193" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:31.514281 9841 mvcc.cc:205] Tried to move back new op lower bound from 7298406856746344448 to 7298406856608464896. Current Snapshot: MvccSnapshot[applied={T|T < 7298406856746344448 or (T in {7298406856746344448})}]
I20260619 02:53:31.516031 9361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:31.516119 9361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:53:31.516819 9361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3f1d31d0c7546b4b45f13a813ba53aa: Sending DeleteTablet for 3 replicas of tablet 870afcbb7c014bb9b178a96ab10a5943
I20260619 02:53:31.517146 9616 tablet_service.cc:1514] Processing DeleteTablet for tablet 870afcbb7c014bb9b178a96ab10a5943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:31 UTC) from {username='slave'} at 127.0.0.1:51696
I20260619 02:53:31.517159 9485 tablet_service.cc:1514] Processing DeleteTablet for tablet 870afcbb7c014bb9b178a96ab10a5943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:31 UTC) from {username='slave'} at 127.0.0.1:40238
I20260619 02:53:31.517349 9747 tablet_service.cc:1514] Processing DeleteTablet for tablet 870afcbb7c014bb9b178a96ab10a5943 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:53:31 UTC) from {username='slave'} at 127.0.0.1:35554
I20260619 02:53:31.517536 9852 tablet_replica.cc:330] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: stopping tablet replica
I20260619 02:53:31.517549 9850 tablet_replica.cc:330] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97: stopping tablet replica
I20260619 02:53:31.517632 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 9423
I20260619 02:53:31.519299 9852 raft_consensus.cc:2229] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:53:31.519426 9852 raft_consensus.cc:2258] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:31.519817 9852 ts_tablet_manager.cc:1905] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:53:31.521543 9852 ts_tablet_manager.cc:1918] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:53:31.521615 9852 log.cc:1196] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: Deleting WAL directory at /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837375929108-22163-0/minicluster-data/ts-2/wal/wals/870afcbb7c014bb9b178a96ab10a5943
I20260619 02:53:31.521828 9852 ts_tablet_manager.cc:1939] T 870afcbb7c014bb9b178a96ab10a5943 P 4021bb31d81b4e0db67a54ca39383a7e: Deleting consensus metadata
I20260619 02:53:31.522224 9348 catalog_manager.cc:4841] TS 4021bb31d81b4e0db67a54ca39383a7e (127.21.164.195:34791): tablet 870afcbb7c014bb9b178a96ab10a5943 (table test-table [id=13dd85424df84a89b4aa4844f7417309]) successfully deleted
W20260619 02:53:31.522913 9348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.164.193:33629 (error 108)
I20260619 02:53:31.522984 9850 raft_consensus.cc:2229] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:53:31.523124 9850 raft_consensus.cc:2258] T 870afcbb7c014bb9b178a96ab10a5943 P 436e34042a9d4adc961adf44c746ad97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:53:31.523483 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 9554
I20260619 02:53:31.523531 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.523409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.193:33629 (local address 127.0.0.1:40248)
0619 02:53:31.523455 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.523495 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.193:33629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 02:53:31.523630 9348 catalog_manager.cc:4584] TS a11a1931381c409b950f7a50e8c7c490 (127.21.164.193:33629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 870afcbb7c014bb9b178a96ab10a5943: Network error: Client connection negotiation failed: client connection to 127.21.164.193:33629: connect: Connection refused (error 111)
I20260619 02:53:31.528595 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 9685
I20260619 02:53:31.528618 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:31.528508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.194:44629 (local address 127.0.0.1:51702)
0619 02:53:31.528559 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:31.528583 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.194:44629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 02:53:31.532967 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 9332
2026-06-19T02:53:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (853 ms)
[----------] 6 tests from EncryptionPolicyTest (8242 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:31 dist-test-slave-jt0t krb5kdc[9856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:31 dist-test-slave-jt0t krb5kdc[9856](info): set up 2 sockets
Jun 19 02:53:31 dist-test-slave-jt0t krb5kdc[9856](info): commencing operation
krb5kdc: starting...
W20260619 02:53:32.484333 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:32 dist-test-slave-jt0t krb5kdc[9856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:32.599504 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:32.679051 9869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:32.679277 9869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:32.679301 9869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:32.680451 9869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:32.680490 9869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:32.680502 9869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:32.680513 9869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:32.680524 9869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:32.682088 9869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38747
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:32.682294 9869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:32.682475 9869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:32.687186 9877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:32.687204 9869 server_base.cc:1034] running on GCE node
W20260619 02:53:32.687171 9874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:32.687160 9875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:32.687640 9869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:32.687914 9869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:32.689091 9869 hybrid_clock.cc:648] HybridClock initialized: now 1781837612689073 us; error 29 us; skew 500 ppm
Jun 19 02:53:32 dist-test-slave-jt0t krb5kdc[9856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:32.691561 9869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:32.691956 9869 webserver.cc:460] Webserver started at http://127.0.0.1:34927/ using document root <none> and password file <none>
I20260619 02:53:32.692168 9869 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:32.692216 9869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:32.692335 9869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:32.693405 9869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "199c2e1b8dd14e69a4be1cdbc5ff4efd"
format_stamp: "Formatted at 2026-06-19 02:53:32 on dist-test-slave-jt0t"
server_key: "592bbb5d29873164f6dc4ee4f5e204ba"
server_key_iv: "949bd927682379b4ddfaf1a7211cee5b"
server_key_version: "encryptionkey@0"
I20260619 02:53:32.693729 9869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "199c2e1b8dd14e69a4be1cdbc5ff4efd"
format_stamp: "Formatted at 2026-06-19 02:53:32 on dist-test-slave-jt0t"
server_key: "592bbb5d29873164f6dc4ee4f5e204ba"
server_key_iv: "949bd927682379b4ddfaf1a7211cee5b"
server_key_version: "encryptionkey@0"
I20260619 02:53:32.694948 9869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:32.695749 9884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:32.695935 9869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:32.696051 9869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "199c2e1b8dd14e69a4be1cdbc5ff4efd"
format_stamp: "Formatted at 2026-06-19 02:53:32 on dist-test-slave-jt0t"
server_key: "592bbb5d29873164f6dc4ee4f5e204ba"
server_key_iv: "949bd927682379b4ddfaf1a7211cee5b"
server_key_version: "encryptionkey@0"
I20260619 02:53:32.696110 9869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:32.726692 9869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:32.729018 9869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:32.729154 9869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:32.732748 9869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38747
I20260619 02:53:32.732796 9936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38747 every 8 connection(s)
I20260619 02:53:32.733125 9869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:32.733911 9937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:32.734338 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 9869
I20260619 02:53:32.734441 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:32.734627 22163 external_mini_cluster.cc:1428] Setting key 7301917703ad1b4edcf664cedfc82e90
I20260619 02:53:32.736778 9937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd: Bootstrap starting.
I20260619 02:53:32.737317 9937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:32.737581 9937 log.cc:824] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:32.738164 9937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd: No bootstrap required, opened a new log
I20260619 02:53:32.739912 9937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199c2e1b8dd14e69a4be1cdbc5ff4efd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38747 } }
I20260619 02:53:32.740046 9937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:32.740067 9937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 199c2e1b8dd14e69a4be1cdbc5ff4efd, State: Initialized, Role: FOLLOWER
I20260619 02:53:32.740171 9937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199c2e1b8dd14e69a4be1cdbc5ff4efd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38747 } }
I20260619 02:53:32.740238 9937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:32.740260 9937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:32.740305 9937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [term 0 FOLLOWER]: Advancing to term 1
Jun 19 02:53:32 dist-test-slave-jt0t krb5kdc[9856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:53:32.740904 9937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199c2e1b8dd14e69a4be1cdbc5ff4efd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38747 } }
I20260619 02:53:32.741014 9937 leader_election.cc:302] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 199c2e1b8dd14e69a4be1cdbc5ff4efd; no voters:
I20260619 02:53:32.741189 9937 leader_election.cc:288] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:32.741232 9942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:32.741391 9942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [term 1 LEADER]: Becoming Leader. State: Replica: 199c2e1b8dd14e69a4be1cdbc5ff4efd, State: Running, Role: LEADER
I20260619 02:53:32.741444 9937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:32.741479 9942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199c2e1b8dd14e69a4be1cdbc5ff4efd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38747 } }
I20260619 02:53:32.741825 9942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 199c2e1b8dd14e69a4be1cdbc5ff4efd. Latest consensus state: current_term: 1 leader_uuid: "199c2e1b8dd14e69a4be1cdbc5ff4efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199c2e1b8dd14e69a4be1cdbc5ff4efd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38747 } } }
I20260619 02:53:32.741907 9942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:32.741837 9944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "199c2e1b8dd14e69a4be1cdbc5ff4efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199c2e1b8dd14e69a4be1cdbc5ff4efd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38747 } } }
I20260619 02:53:32.742084 9944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:32.742107 9949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:32.742642 9949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:32.744089 9949 catalog_manager.cc:1347] Generated new cluster ID: 1ee04fec2b5648f190342551794d63fd
I20260619 02:53:32.744138 9949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:32.745272 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:32.735156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59891 (local address 127.0.0.1:38747)
0619 02:53:32.735313 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:32.735321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:32.735345 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:53:32.735744 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:32.735750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:32.735794 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:32.735904 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:32.736689 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:32.737652 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:32.738404 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:32.738806 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:32.738898 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:32.741222 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:32.741226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:32.741227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:32.741240 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:32.743204 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:32.743630 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:32.743633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:32.743633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:32.743735 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:32.743980 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:32.743981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:32.743981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:32.744359 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:32.744875 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:32.744968 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:53:32.745081 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260619 02:53:32.746357 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:32.757715 9949 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:32.758383 9949 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:32.759760 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:32.747737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38325 (local address 127.0.0.1:38747)
0619 02:53:32.747771 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:32.747772 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:32.752822 (+ 5050us) server_negotiation.cc:407] Connection header received
0619 02:53:32.752899 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:32.752900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:32.752912 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:32.752966 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:32.753566 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:32.754358 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:32.755305 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:32.755662 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:32.755728 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:32.757087 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:32.757089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:32.757089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:32.757097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:32.758670 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:32.759051 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:32.759052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:32.759053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:32.759111 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:32.759297 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:32.759298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:32.759298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:32.759415 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:32.759511 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:32.759609 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 02:53:32.759670 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:53:32.765654 9949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 199c2e1b8dd14e69a4be1cdbc5ff4efd: Generated new TSK 0
I20260619 02:53:32.765816 9949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:32.768792 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 9869
W20260619 02:53:32.772372 9854 connection.cc:535] client connection to 127.0.0.1:38747 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38747): Connection reset by peer (error 104)
2026-06-19T02:53:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1234 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:32 dist-test-slave-jt0t krb5kdc[9968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:32 dist-test-slave-jt0t krb5kdc[9968](info): set up 2 sockets
Jun 19 02:53:32 dist-test-slave-jt0t krb5kdc[9968](info): commencing operation
krb5kdc: starting...
W20260619 02:53:33.712437 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:33 dist-test-slave-jt0t krb5kdc[9968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:33.827308 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45803
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:33.907465 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:33.907698 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:33.907724 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:33.908954 9981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:33.908996 9981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:33.909009 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:33.909020 9981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:33.909031 9981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:33.910652 9981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45803
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:33.910908 9981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:33.911119 9981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:33.915798 9986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:33.915798 9987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:33.915832 9981 server_base.cc:1034] running on GCE node
W20260619 02:53:33.915809 9989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:33.916213 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:33.916513 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:33.917658 9981 hybrid_clock.cc:648] HybridClock initialized: now 1781837613917648 us; error 35 us; skew 500 ppm
Jun 19 02:53:33 dist-test-slave-jt0t krb5kdc[9968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:33.925243 9981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:33.925709 9981 webserver.cc:460] Webserver started at http://127.0.0.1:44307/ using document root <none> and password file <none>
I20260619 02:53:33.925922 9981 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:33.925971 9981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:33.926096 9981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:33.927127 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "1f8e6b190f3e4a7db7164157517705da"
format_stamp: "Formatted at 2026-06-19 02:53:33 on dist-test-slave-jt0t"
server_key: "1aebddd9a590c02aad26790335a68fae"
server_key_iv: "3b3ca49a91ec9c1ef3ca63ea439bf021"
server_key_version: "encryptionkey@0"
I20260619 02:53:33.927449 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "1f8e6b190f3e4a7db7164157517705da"
format_stamp: "Formatted at 2026-06-19 02:53:33 on dist-test-slave-jt0t"
server_key: "1aebddd9a590c02aad26790335a68fae"
server_key_iv: "3b3ca49a91ec9c1ef3ca63ea439bf021"
server_key_version: "encryptionkey@0"
I20260619 02:53:33.928712 9981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:33.929400 9996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:33.929548 9981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:33.929632 9981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "1f8e6b190f3e4a7db7164157517705da"
format_stamp: "Formatted at 2026-06-19 02:53:33 on dist-test-slave-jt0t"
server_key: "1aebddd9a590c02aad26790335a68fae"
server_key_iv: "3b3ca49a91ec9c1ef3ca63ea439bf021"
server_key_version: "encryptionkey@0"
I20260619 02:53:33.929698 9981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:33.952896 9981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:33.955299 9981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:33.955457 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:33.960335 9981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45803
I20260619 02:53:33.960376 10048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45803 every 8 connection(s)
I20260619 02:53:33.960829 9981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:33.961721 10049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:33.962026 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 9981
I20260619 02:53:33.962149 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:33.962333 22163 external_mini_cluster.cc:1428] Setting key 30c1f7f38fbaea00870c53291f8ca584
I20260619 02:53:33.964439 10049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da: Bootstrap starting.
I20260619 02:53:33.965039 10049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:33.965301 10049 log.cc:824] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:33.965970 10049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da: No bootstrap required, opened a new log
I20260619 02:53:33.968107 10049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8e6b190f3e4a7db7164157517705da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260619 02:53:33.968279 10049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:33.968320 10049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8e6b190f3e4a7db7164157517705da, State: Initialized, Role: FOLLOWER
I20260619 02:53:33.968444 10049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8e6b190f3e4a7db7164157517705da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260619 02:53:33.968516 10049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:33.968545 10049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:33.968596 10049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:33.969180 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:33.962855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59709 (local address 127.0.0.1:45803)
0619 02:53:33.963067 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:33.963078 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:33.963113 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:53:33.963678 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:33.963687 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:33.963753 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:33.963909 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:33.964179 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:33.965656 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:33.966737 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:33.967393 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:33.967525 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:33.968285 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:33.968290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:33.968290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:33.968509 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:33.968551 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:33.968566 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:33.968670 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:33.968930 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 02:53:33.968962 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260619 02:53:33.969184 10049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8e6b190f3e4a7db7164157517705da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260619 02:53:33.969317 10049 leader_election.cc:302] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f8e6b190f3e4a7db7164157517705da; no voters:
I20260619 02:53:33.969643 10049 leader_election.cc:288] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:33.969727 10054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:33.969892 10054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [term 1 LEADER]: Becoming Leader. State: Replica: 1f8e6b190f3e4a7db7164157517705da, State: Running, Role: LEADER
I20260619 02:53:33.970044 10049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:33.970016 10054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8e6b190f3e4a7db7164157517705da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } }
I20260619 02:53:33.970144 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:33.970465 10054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f8e6b190f3e4a7db7164157517705da. Latest consensus state: current_term: 1 leader_uuid: "1f8e6b190f3e4a7db7164157517705da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8e6b190f3e4a7db7164157517705da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } } }
I20260619 02:53:33.970599 10054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:33.971704 10055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f8e6b190f3e4a7db7164157517705da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8e6b190f3e4a7db7164157517705da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45803 } } }
W20260619 02:53:33.971769 10072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 02:53:33.971786 10055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da [sys.catalog]: This master's current role is: LEADER
W20260619 02:53:33.971921 10072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:33.972160 10063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:33.972344 10063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:33.973970 10063 catalog_manager.cc:1347] Generated new cluster ID: 635644c3ffcf45d6b191a419bcda67ed
I20260619 02:53:33.974025 10063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:33.975512 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:33.971674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57367 (local address 127.0.0.1:45803)
0619 02:53:33.971813 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:33.971814 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:33.971822 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:33.971892 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:33.971893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:33.971903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:33.971958 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:33.972154 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:33.972964 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:33.973941 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:33.974313 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:33.974374 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:33.975152 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:33.975153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:33.975154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:33.975159 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:33.975174 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:33.975178 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:33.975223 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:33.975415 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 02:53:33.975425 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260619 02:53:34.000167 10063 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:34.000948 10063 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:34.006690 10063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f8e6b190f3e4a7db7164157517705da: Generated new TSK 0
I20260619 02:53:34.006896 10063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:34.009095 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 9981
W20260619 02:53:34.012902 9966 connection.cc:535] client connection to 127.0.0.1:45803 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45803): Connection reset by peer (error 104)
2026-06-19T02:53:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1240 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:34 dist-test-slave-jt0t krb5kdc[10080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:34 dist-test-slave-jt0t krb5kdc[10080](info): set up 2 sockets
Jun 19 02:53:34 dist-test-slave-jt0t krb5kdc[10080](info): commencing operation
krb5kdc: starting...
W20260619 02:53:34.964352 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:35 dist-test-slave-jt0t krb5kdc[10080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:35.077876 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:35.158982 10093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:35.159204 10093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:35.159232 10093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:35.160391 10093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:35.160434 10093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:35.160454 10093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:35.160468 10093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:35.160485 10093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:35.162056 10093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38953
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:35.162297 10093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:35.162511 10093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:53:35.167208 10093 server_base.cc:1034] running on GCE node
W20260619 02:53:35.167194 10101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:35.167330 10099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:35.167234 10098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:35.167743 10093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:35.168069 10093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:35.169217 10093 hybrid_clock.cc:648] HybridClock initialized: now 1781837615169194 us; error 33 us; skew 500 ppm
Jun 19 02:53:35 dist-test-slave-jt0t krb5kdc[10080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:35.171571 10093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:35.172055 10093 webserver.cc:460] Webserver started at http://127.0.0.1:43213/ using document root <none> and password file <none>
I20260619 02:53:35.172266 10093 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:35.172307 10093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:35.172428 10093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:35.173555 10093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "46f8b59f0c5f404298d4801a4c90131f"
format_stamp: "Formatted at 2026-06-19 02:53:35 on dist-test-slave-jt0t"
server_key: "62d14e0999fd7c89b0b603e500b1e191"
server_key_iv: "589447adcc32c3a061d64be8ea14f084"
server_key_version: "encryptionkey@0"
I20260619 02:53:35.173887 10093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "46f8b59f0c5f404298d4801a4c90131f"
format_stamp: "Formatted at 2026-06-19 02:53:35 on dist-test-slave-jt0t"
server_key: "62d14e0999fd7c89b0b603e500b1e191"
server_key_iv: "589447adcc32c3a061d64be8ea14f084"
server_key_version: "encryptionkey@0"
I20260619 02:53:35.175194 10093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:35.175980 10108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:35.176160 10093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:35.176255 10093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "46f8b59f0c5f404298d4801a4c90131f"
format_stamp: "Formatted at 2026-06-19 02:53:35 on dist-test-slave-jt0t"
server_key: "62d14e0999fd7c89b0b603e500b1e191"
server_key_iv: "589447adcc32c3a061d64be8ea14f084"
server_key_version: "encryptionkey@0"
I20260619 02:53:35.176316 10093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:35.179587 10093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:35.181641 10093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:35.181756 10093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:35.185361 10093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38953
I20260619 02:53:35.185393 10160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38953 every 8 connection(s)
I20260619 02:53:35.185698 10093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:35.186643 10161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:35.188899 10161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f: Bootstrap starting.
I20260619 02:53:35.189433 10161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:35.189678 10161 log.cc:824] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:35.190348 10161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f: No bootstrap required, opened a new log
I20260619 02:53:35.191902 10161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f8b59f0c5f404298d4801a4c90131f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } }
I20260619 02:53:35.192049 10161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:35.192080 10161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f8b59f0c5f404298d4801a4c90131f, State: Initialized, Role: FOLLOWER
I20260619 02:53:35.192233 10161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f8b59f0c5f404298d4801a4c90131f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } }
I20260619 02:53:35.192308 10161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:35.192338 10161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:35.192332 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 10093
I20260619 02:53:35.192385 10161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:35.192454 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:35.192628 22163 external_mini_cluster.cc:1428] Setting key 48fb6423b3d756a39a9c29cf2a9bcbbb
I20260619 02:53:35.193017 10161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f8b59f0c5f404298d4801a4c90131f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } }
I20260619 02:53:35.193121 10161 leader_election.cc:302] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46f8b59f0c5f404298d4801a4c90131f; no voters:
I20260619 02:53:35.193264 10161 leader_election.cc:288] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:35.193295 10166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:35.193467 10166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [term 1 LEADER]: Becoming Leader. State: Replica: 46f8b59f0c5f404298d4801a4c90131f, State: Running, Role: LEADER
I20260619 02:53:35.193562 10161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:35.193579 10166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f8b59f0c5f404298d4801a4c90131f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } }
I20260619 02:53:35.194013 10166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46f8b59f0c5f404298d4801a4c90131f. Latest consensus state: current_term: 1 leader_uuid: "46f8b59f0c5f404298d4801a4c90131f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f8b59f0c5f404298d4801a4c90131f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } } }
I20260619 02:53:35.194118 10166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:35.194536 10171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:35.194785 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:35.193108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54371 (local address 127.0.0.1:38953)
0619 02:53:35.193302 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:35.193312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:35.193346 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:53:35.193479 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:35.193487 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:35.193516 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:35.193655 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:35.194277 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:35.194281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:35.194281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:35.194285 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:35.194326 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:35.194342 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:35.194415 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:35.194553 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 02:53:35.194597 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260619 02:53:35.195042 10171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:35.194339 10167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46f8b59f0c5f404298d4801a4c90131f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f8b59f0c5f404298d4801a4c90131f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } } }
I20260619 02:53:35.195215 10167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:35.195681 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:35.196722 10171 catalog_manager.cc:1347] Generated new cluster ID: 473c2117b8a14c7497adbc2d3aeb8482
I20260619 02:53:35.196767 10171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:35.198153 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:35.197062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52789 (local address 127.0.0.1:38953)
0619 02:53:35.197092 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:35.197093 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:35.197336 (+ 243us) server_negotiation.cc:407] Connection header received
0619 02:53:35.197406 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:35.197406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:35.197412 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:35.197468 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:35.197962 (+ 494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:35.197963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:35.197963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:35.197967 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:35.197982 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:35.197986 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:35.198030 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:35.198078 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 02:53:35.198091 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 02:53:35.236718 10171 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:35.236948 10171 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:35.244346 10171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46f8b59f0c5f404298d4801a4c90131f: Generated new TSK 0
I20260619 02:53:35.244539 10171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:35.251209 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 10093
2026-06-19T02:53:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1241 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:35 dist-test-slave-jt0t krb5kdc[10192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:35 dist-test-slave-jt0t krb5kdc[10192](info): set up 2 sockets
Jun 19 02:53:35 dist-test-slave-jt0t krb5kdc[10192](info): commencing operation
krb5kdc: starting...
W20260619 02:53:36.196393 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:36 dist-test-slave-jt0t krb5kdc[10192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:36.307237 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:36.385776 10205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:36.385989 10205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:36.386019 10205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:36.387192 10205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:36.387235 10205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:36.387253 10205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:36.387270 10205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:36.387285 10205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:36.388788 10205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:36.389034 10205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:36.389236 10205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:36.393894 10210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:36.393889 10211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:36.393889 10213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:36.394134 10205 server_base.cc:1034] running on GCE node
I20260619 02:53:36.394378 10205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:36.394658 10205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:36.395814 10205 hybrid_clock.cc:648] HybridClock initialized: now 1781837616395797 us; error 26 us; skew 500 ppm
Jun 19 02:53:36 dist-test-slave-jt0t krb5kdc[10192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:36.398247 10205 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:36.398680 10205 webserver.cc:460] Webserver started at http://127.0.0.1:36233/ using document root <none> and password file <none>
I20260619 02:53:36.398886 10205 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:36.398931 10205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:36.399057 10205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:36.400065 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "822b2f8efecc4991828b902f30b067e7"
format_stamp: "Formatted at 2026-06-19 02:53:36 on dist-test-slave-jt0t"
server_key: "dd5669865f4590194735d960fb5f1d39"
server_key_iv: "b53067e146ef761251db7467700f4bd0"
server_key_version: "encryptionkey@0"
I20260619 02:53:36.400385 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "822b2f8efecc4991828b902f30b067e7"
format_stamp: "Formatted at 2026-06-19 02:53:36 on dist-test-slave-jt0t"
server_key: "dd5669865f4590194735d960fb5f1d39"
server_key_iv: "b53067e146ef761251db7467700f4bd0"
server_key_version: "encryptionkey@0"
I20260619 02:53:36.401700 10205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:53:36.402433 10220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:36.402618 10205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:53:36.402700 10205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "822b2f8efecc4991828b902f30b067e7"
format_stamp: "Formatted at 2026-06-19 02:53:36 on dist-test-slave-jt0t"
server_key: "dd5669865f4590194735d960fb5f1d39"
server_key_iv: "b53067e146ef761251db7467700f4bd0"
server_key_version: "encryptionkey@0"
I20260619 02:53:36.402762 10205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:36.427142 10205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:36.429255 10205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:36.429404 10205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:36.433063 10205 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36953
I20260619 02:53:36.433132 10272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36953 every 8 connection(s)
I20260619 02:53:36.433439 10205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:36.434283 10273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:36.436458 10273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7: Bootstrap starting.
I20260619 02:53:36.437043 10273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:36.437294 10273 log.cc:824] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:36.437922 10273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7: No bootstrap required, opened a new log
I20260619 02:53:36.439445 10273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822b2f8efecc4991828b902f30b067e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260619 02:53:36.439585 10273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:36.439607 10273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822b2f8efecc4991828b902f30b067e7, State: Initialized, Role: FOLLOWER
I20260619 02:53:36.439735 10273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822b2f8efecc4991828b902f30b067e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260619 02:53:36.439796 10273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:36.439815 10273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:36.439844 10273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:36.440333 10273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822b2f8efecc4991828b902f30b067e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260619 02:53:36.440415 10273 leader_election.cc:302] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 822b2f8efecc4991828b902f30b067e7; no voters:
I20260619 02:53:36.440577 10273 leader_election.cc:288] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:36.440621 10276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:36.440806 10276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [term 1 LEADER]: Becoming Leader. State: Replica: 822b2f8efecc4991828b902f30b067e7, State: Running, Role: LEADER
I20260619 02:53:36.440843 10273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:36.440935 10276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822b2f8efecc4991828b902f30b067e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260619 02:53:36.441289 10278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 822b2f8efecc4991828b902f30b067e7. Latest consensus state: current_term: 1 leader_uuid: "822b2f8efecc4991828b902f30b067e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822b2f8efecc4991828b902f30b067e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } } }
I20260619 02:53:36.441285 10277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "822b2f8efecc4991828b902f30b067e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822b2f8efecc4991828b902f30b067e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } } }
I20260619 02:53:36.441377 10278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:36.441383 10277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:36.441558 10280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:36.442036 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 10205
I20260619 02:53:36.442068 10280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:36.442152 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:36.442324 22163 external_mini_cluster.cc:1428] Setting key f77c43ac756fba336d1ff34ad1753713
I20260619 02:53:36.443967 10280 catalog_manager.cc:1347] Generated new cluster ID: fb43879f3584422e8ebe995c0486233c
I20260619 02:53:36.444029 10280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:53:36 dist-test-slave-jt0t krb5kdc[10192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:53:36.451634 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:36.442812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48431 (local address 127.0.0.1:36953)
0619 02:53:36.442998 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:36.443008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:36.443039 (+ 31us) server_negotiation.cc:407] Connection header received
0619 02:53:36.443154 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:36.443161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:36.443219 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:36.443367 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:36.444134 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:36.445134 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:36.445861 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:36.446309 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:36.446418 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:36.448591 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:36.448596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:36.448596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:36.448608 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:36.450183 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:36.450504 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:36.450508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:36.450508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:36.450618 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:36.450784 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:36.450784 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:36.450785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:36.451038 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:36.451172 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:36.451278 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 02:53:36.451418 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260619 02:53:36.452711 10280 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:36.452706 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:36.453415 10280 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:36.460775 10280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 822b2f8efecc4991828b902f30b067e7: Generated new TSK 0
I20260619 02:53:36.461046 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:36.454109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55151 (local address 127.0.0.1:36953)
0619 02:53:36.454143 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:36.454144 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:36.454214 (+ 70us) server_negotiation.cc:407] Connection header received
0619 02:53:36.454278 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:36.454279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:36.454288 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:36.454335 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:36.454882 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:36.455658 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:36.456673 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:36.457003 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:36.457063 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:36.458316 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:36.458317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:36.458317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:36.458324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:36.459817 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:36.460156 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:36.460157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:36.460157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:36.460215 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:36.460419 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:36.460420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:36.460420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:36.460545 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:36.460675 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:36.460887 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 02:53:36.460969 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":15}
I20260619 02:53:36.461382 10280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:36.468713 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 10205
2026-06-19T02:53:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:36 dist-test-slave-jt0t krb5kdc[10304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:36 dist-test-slave-jt0t krb5kdc[10304](info): set up 2 sockets
Jun 19 02:53:36 dist-test-slave-jt0t krb5kdc[10304](info): commencing operation
krb5kdc: starting...
W20260619 02:53:37.424862 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:37 dist-test-slave-jt0t krb5kdc[10304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:37.541572 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:37.621327 10317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:37.621546 10317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:37.621569 10317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:37.622687 10317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:37.622720 10317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:37.622733 10317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:37.622743 10317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:37.622754 10317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:37.624169 10317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:37.624364 10317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:37.624543 10317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:37.629208 10323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:37.629352 10317 server_base.cc:1034] running on GCE node
W20260619 02:53:37.629205 10325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:37.629323 10322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:37.629704 10317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:37.629987 10317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:37.631139 10317 hybrid_clock.cc:648] HybridClock initialized: now 1781837617631118 us; error 31 us; skew 500 ppm
Jun 19 02:53:37 dist-test-slave-jt0t krb5kdc[10304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:37.633669 10317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:37.634132 10317 webserver.cc:460] Webserver started at http://127.0.0.1:36261/ using document root <none> and password file <none>
I20260619 02:53:37.634320 10317 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:37.634361 10317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:37.634459 10317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:37.635485 10317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "577b197d69e54c5baef1ef8fe783ef82"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-jt0t"
server_key: "b0ff4d451f2f72186188a5a7ea7e4e12"
server_key_iv: "352b2572b5ac22bef8c643b8a906af9c"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.635849 10317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "577b197d69e54c5baef1ef8fe783ef82"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-jt0t"
server_key: "b0ff4d451f2f72186188a5a7ea7e4e12"
server_key_iv: "352b2572b5ac22bef8c643b8a906af9c"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.637154 10317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:37.637892 10332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:37.638075 10317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:37.638159 10317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "577b197d69e54c5baef1ef8fe783ef82"
format_stamp: "Formatted at 2026-06-19 02:53:37 on dist-test-slave-jt0t"
server_key: "b0ff4d451f2f72186188a5a7ea7e4e12"
server_key_iv: "352b2572b5ac22bef8c643b8a906af9c"
server_key_version: "encryptionkey@0"
I20260619 02:53:37.638216 10317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:37.654413 10317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:37.656243 10317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:37.656354 10317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:37.659909 10317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45403
I20260619 02:53:37.659960 10384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45403 every 8 connection(s)
I20260619 02:53:37.660262 10317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:37.661130 10385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:37.663336 10385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82: Bootstrap starting.
I20260619 02:53:37.663934 10385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:37.664183 10385 log.cc:824] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:37.664884 10385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82: No bootstrap required, opened a new log
I20260619 02:53:37.666325 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 10317
I20260619 02:53:37.666441 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:37.666599 22163 external_mini_cluster.cc:1428] Setting key 9ad5676f350558324ba28f8dc0546438
I20260619 02:53:37.666503 10385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b197d69e54c5baef1ef8fe783ef82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260619 02:53:37.666638 10385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:37.666678 10385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577b197d69e54c5baef1ef8fe783ef82, State: Initialized, Role: FOLLOWER
I20260619 02:53:37.666822 10385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b197d69e54c5baef1ef8fe783ef82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260619 02:53:37.666903 10385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:37.667207 10385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:37.667272 10385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:37.668378 10385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b197d69e54c5baef1ef8fe783ef82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260619 02:53:37.668488 10385 leader_election.cc:302] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 577b197d69e54c5baef1ef8fe783ef82; no voters:
I20260619 02:53:37.668663 10385 leader_election.cc:288] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:37.668710 10390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:37.668850 10390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [term 1 LEADER]: Becoming Leader. State: Replica: 577b197d69e54c5baef1ef8fe783ef82, State: Running, Role: LEADER
I20260619 02:53:37.668897 10385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:37.669003 10390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b197d69e54c5baef1ef8fe783ef82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260619 02:53:37.669380 10390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 577b197d69e54c5baef1ef8fe783ef82. Latest consensus state: current_term: 1 leader_uuid: "577b197d69e54c5baef1ef8fe783ef82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b197d69e54c5baef1ef8fe783ef82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } } }
I20260619 02:53:37.669580 10390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:37.669512 10391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "577b197d69e54c5baef1ef8fe783ef82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577b197d69e54c5baef1ef8fe783ef82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } } }
I20260619 02:53:37.669802 10391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:37.670137 10399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:37.670611 10399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:37.672411 10399 catalog_manager.cc:1347] Generated new cluster ID: 39db3f39bb354b6799d8995669acb492
I20260619 02:53:37.672456 10399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:37.672683 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.667068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38877 (local address 127.0.0.1:45403)
0619 02:53:37.667241 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.667252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.667284 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:53:37.667411 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.667418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.667483 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.667632 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:37.667818 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.669190 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.670286 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.670886 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.670986 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.672016 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:37.672020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:37.672020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:37.672202 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:37.672232 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:37.672243 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:37.672299 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.672383 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.672445 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260619 02:53:37.673470 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:37.683773 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:37.680983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37709 (local address 127.0.0.1:45403)
0619 02:53:37.681029 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:37.681031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:37.681240 (+ 209us) server_negotiation.cc:407] Connection header received
0619 02:53:37.681303 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:37.681304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:37.681316 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:37.681370 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:37.681500 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.681967 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.682677 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:37.682873 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:37.682927 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:37.683529 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:37.683530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:37.683531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:37.683536 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:37.683551 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:37.683555 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:37.683601 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:37.683659 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 02:53:37.683685 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 02:53:37.703291 10399 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:37.704046 10399 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:37.707646 10399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 577b197d69e54c5baef1ef8fe783ef82: Generated new TSK 0
I20260619 02:53:37.707837 10399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:37.716091 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 10317
2026-06-19T02:53:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:37 dist-test-slave-jt0t krb5kdc[10416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:37 dist-test-slave-jt0t krb5kdc[10416](info): set up 2 sockets
Jun 19 02:53:37 dist-test-slave-jt0t krb5kdc[10416](info): commencing operation
krb5kdc: starting...
W20260619 02:53:38.672437 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:38 dist-test-slave-jt0t krb5kdc[10416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:38.785535 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:38.865191 10429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:38.865408 10429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:38.865438 10429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:38.866694 10429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:38.866739 10429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:38.866758 10429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:38.866775 10429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:38.866797 10429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:38.868338 10429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:38.868593 10429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:38.868830 10429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:38.873524 10434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:38.873544 10437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:38.873524 10435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:38.873564 10429 server_base.cc:1034] running on GCE node
I20260619 02:53:38.873966 10429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:38.874272 10429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:38.875429 10429 hybrid_clock.cc:648] HybridClock initialized: now 1781837618875410 us; error 32 us; skew 500 ppm
Jun 19 02:53:38 dist-test-slave-jt0t krb5kdc[10416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:38.877828 10429 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:38.878284 10429 webserver.cc:460] Webserver started at http://127.0.0.1:45583/ using document root <none> and password file <none>
I20260619 02:53:38.878477 10429 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:38.878525 10429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:38.878610 10429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:38.879613 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "3584753823c44fbbaa56f4808f8ee84c"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-jt0t"
server_key: "e70be2a45cbe76d8e3fac4f6fb3a0a8f"
server_key_iv: "277dd6ca76d31a0744b2d6f82266fca6"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.879899 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "3584753823c44fbbaa56f4808f8ee84c"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-jt0t"
server_key: "e70be2a45cbe76d8e3fac4f6fb3a0a8f"
server_key_iv: "277dd6ca76d31a0744b2d6f82266fca6"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.881161 10429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:53:38.881839 10444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:38.881999 10429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:53:38.882074 10429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "3584753823c44fbbaa56f4808f8ee84c"
format_stamp: "Formatted at 2026-06-19 02:53:38 on dist-test-slave-jt0t"
server_key: "e70be2a45cbe76d8e3fac4f6fb3a0a8f"
server_key_iv: "277dd6ca76d31a0744b2d6f82266fca6"
server_key_version: "encryptionkey@0"
I20260619 02:53:38.882145 10429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:38.885424 10429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:38.887243 10429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:38.887365 10429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:38.891086 10429 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36445
I20260619 02:53:38.891116 10496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36445 every 8 connection(s)
I20260619 02:53:38.891425 10429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:38.892349 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:38.894555 10497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c: Bootstrap starting.
I20260619 02:53:38.895063 10497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:38.895283 10497 log.cc:824] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:38.895897 10497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c: No bootstrap required, opened a new log
I20260619 02:53:38.897497 10497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584753823c44fbbaa56f4808f8ee84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36445 } }
I20260619 02:53:38.897635 10497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:38.897656 10497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3584753823c44fbbaa56f4808f8ee84c, State: Initialized, Role: FOLLOWER
I20260619 02:53:38.897766 10497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584753823c44fbbaa56f4808f8ee84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36445 } }
I20260619 02:53:38.897836 10497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:38.897861 10497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:38.897892 10497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:38.898382 10497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584753823c44fbbaa56f4808f8ee84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36445 } }
I20260619 02:53:38.898466 10497 leader_election.cc:302] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3584753823c44fbbaa56f4808f8ee84c; no voters:
I20260619 02:53:38.898602 10497 leader_election.cc:288] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:38.898674 10500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:38.898823 10497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:38.898876 10500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [term 1 LEADER]: Becoming Leader. State: Replica: 3584753823c44fbbaa56f4808f8ee84c, State: Running, Role: LEADER
I20260619 02:53:38.898995 10500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584753823c44fbbaa56f4808f8ee84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36445 } }
I20260619 02:53:38.899549 10502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3584753823c44fbbaa56f4808f8ee84c. Latest consensus state: current_term: 1 leader_uuid: "3584753823c44fbbaa56f4808f8ee84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584753823c44fbbaa56f4808f8ee84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36445 } } }
I20260619 02:53:38.899633 10502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:38.899559 10501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3584753823c44fbbaa56f4808f8ee84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3584753823c44fbbaa56f4808f8ee84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36445 } } }
I20260619 02:53:38.899703 10501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:38.899976 10510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:38.900050 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 10429
I20260619 02:53:38.900161 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:38.900327 22163 external_mini_cluster.cc:1428] Setting key cd21c88e76945cf2c9d0eedcd11020a5
I20260619 02:53:38.900534 10510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:38.902304 10510 catalog_manager.cc:1347] Generated new cluster ID: 3d1ce5d69235482f8aaa4248b4850464
I20260619 02:53:38.902354 10510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:38.902848 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.900963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39529 (local address 127.0.0.1:36445)
0619 02:53:38.901133 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.901142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.901174 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:53:38.901361 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.901369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.901402 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.901537 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:38.902309 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:38.902312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:38.902312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:38.902317 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:38.902368 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:38.902383 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:38.902613 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.902640 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.902678 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260619 02:53:38.905584 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:38.908037 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:38.907031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57941 (local address 127.0.0.1:36445)
0619 02:53:38.907136 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:38.907142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:38.907170 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:53:38.907245 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:38.907246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:38.907251 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:38.907292 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:38.907849 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:38.907850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:38.907850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:38.907854 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:38.907868 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:38.907873 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:38.907914 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:38.907965 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 02:53:38.907976 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:53:38.911722 10510 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:38.911899 10510 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:38.919958 10510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3584753823c44fbbaa56f4808f8ee84c: Generated new TSK 0
I20260619 02:53:38.920181 10510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:38.922127 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 10429
2026-06-19T02:53:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:38 dist-test-slave-jt0t krb5kdc[10528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:38 dist-test-slave-jt0t krb5kdc[10528](info): set up 2 sockets
Jun 19 02:53:38 dist-test-slave-jt0t krb5kdc[10528](info): commencing operation
krb5kdc: starting...
W20260619 02:53:39.876474 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:39 dist-test-slave-jt0t krb5kdc[10528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:39.993819 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:40.074165 10541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:40.074398 10541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:40.074430 10541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:40.075629 10541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:40.075675 10541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:40.075693 10541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:40.075709 10541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:40.075723 10541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:40.077286 10541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:40.077553 10541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:40.077770 10541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:40.082609 10549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:40.082804 10541 server_base.cc:1034] running on GCE node
W20260619 02:53:40.082659 10547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:40.082609 10546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:40.083155 10541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:40.083454 10541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:40.084609 10541 hybrid_clock.cc:648] HybridClock initialized: now 1781837620084595 us; error 27 us; skew 500 ppm
Jun 19 02:53:40 dist-test-slave-jt0t krb5kdc[10528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:40.086977 10541 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:40.087347 10541 webserver.cc:460] Webserver started at http://127.0.0.1:41205/ using document root <none> and password file <none>
I20260619 02:53:40.087531 10541 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:40.087569 10541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:40.087656 10541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:40.088694 10541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "30433e20cb894352bd9067214724d694"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-jt0t"
server_key: "35947f7610e6cf55660ec798afa79522"
server_key_iv: "5f6a6b7d43c6dcb6800ac13beb7fdd76"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.089004 10541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "30433e20cb894352bd9067214724d694"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-jt0t"
server_key: "35947f7610e6cf55660ec798afa79522"
server_key_iv: "5f6a6b7d43c6dcb6800ac13beb7fdd76"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.090165 10541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:40.090786 10556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.090976 10541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:40.091058 10541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "30433e20cb894352bd9067214724d694"
format_stamp: "Formatted at 2026-06-19 02:53:40 on dist-test-slave-jt0t"
server_key: "35947f7610e6cf55660ec798afa79522"
server_key_iv: "5f6a6b7d43c6dcb6800ac13beb7fdd76"
server_key_version: "encryptionkey@0"
I20260619 02:53:40.091112 10541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:40.102175 10541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:40.103839 10541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:40.103956 10541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:40.107606 10541 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40889
I20260619 02:53:40.107646 10608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40889 every 8 connection(s)
I20260619 02:53:40.107963 10541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:40.108557 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 10541
I20260619 02:53:40.108672 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:40.108855 22163 external_mini_cluster.cc:1428] Setting key 1fbe555c3acce57f4c24edb2858dbf08
I20260619 02:53:40.108927 10609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:40.112067 10609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694: Bootstrap starting.
I20260619 02:53:40.112775 10609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:40.113127 10609 log.cc:824] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:40.113855 10609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694: No bootstrap required, opened a new log
Jun 19 02:53:40 dist-test-slave-jt0t krb5kdc[10528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:53:40.115912 10609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30433e20cb894352bd9067214724d694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } }
I20260619 02:53:40.116075 10609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:40.116127 10609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30433e20cb894352bd9067214724d694, State: Initialized, Role: FOLLOWER
I20260619 02:53:40.116261 10609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30433e20cb894352bd9067214724d694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } }
I20260619 02:53:40.116341 10609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:40.116374 10609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:40.116422 10609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:40.117125 10609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30433e20cb894352bd9067214724d694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } }
I20260619 02:53:40.117241 10609 leader_election.cc:302] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30433e20cb894352bd9067214724d694; no voters:
I20260619 02:53:40.117398 10609 leader_election.cc:288] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:40.117440 10614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:40.117584 10614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [term 1 LEADER]: Becoming Leader. State: Replica: 30433e20cb894352bd9067214724d694, State: Running, Role: LEADER
I20260619 02:53:40.117651 10609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:40.117717 10614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30433e20cb894352bd9067214724d694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } }
I20260619 02:53:40.118057 10614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30433e20cb894352bd9067214724d694. Latest consensus state: current_term: 1 leader_uuid: "30433e20cb894352bd9067214724d694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30433e20cb894352bd9067214724d694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } } }
I20260619 02:53:40.118253 10614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:40.118194 10616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30433e20cb894352bd9067214724d694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30433e20cb894352bd9067214724d694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40889 } } }
I20260619 02:53:40.118412 10616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694 [sys.catalog]: This master's current role is: LEADER
W20260619 02:53:40.119388 10629 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:40.119477 10629 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:40.119550 10623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:40.119736 10623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:40.119880 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.109332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:37241 (local address 127.0.0.1:40889)
0619 02:53:40.109513 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.109521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.109547 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:53:40.110044 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.110051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.110104 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.110217 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:40.110982 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.111845 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.112914 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.113483 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.113584 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.116054 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:40.116058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:40.116058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:40.116070 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:40.117936 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.118399 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.118402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.118402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.118541 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.118790 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.118791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.118791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.119104 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:40.119256 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.119584 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.119689 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260619 02:53:40.120978 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:40.122648 10623 catalog_manager.cc:1347] Generated new cluster ID: 69b8fb1960824a128dff7536afdf57d2
I20260619 02:53:40.122700 10623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:40.132085 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:40.122382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:38033 (local address 127.0.0.1:40889)
0619 02:53:40.122416 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:40.122417 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:40.124846 (+ 2429us) server_negotiation.cc:407] Connection header received
0619 02:53:40.124962 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:40.124963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:40.124974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:40.125023 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:40.125670 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.126401 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.127520 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:40.127841 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:40.127914 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:40.129370 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:40.129372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:40.129372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:40.129379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:40.130930 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.131343 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.131343 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.131344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.131401 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:40.131599 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:40.131599 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:40.131599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:40.131713 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:40.131799 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:40.131932 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 02:53:40.131991 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 02:53:40.145035 10623 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:40.145573 10623 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:40.151571 10623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30433e20cb894352bd9067214724d694: Generated new TSK 0
I20260619 02:53:40.151762 10623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:40.159030 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 10541
W20260619 02:53:40.162791 10526 connection.cc:535] client connection to 127.0.0.1:40889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40889): Connection reset by peer (error 104)
2026-06-19T02:53:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:40 dist-test-slave-jt0t krb5kdc[10640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:40 dist-test-slave-jt0t krb5kdc[10640](info): set up 2 sockets
Jun 19 02:53:40 dist-test-slave-jt0t krb5kdc[10640](info): commencing operation
krb5kdc: starting...
W20260619 02:53:41.108408 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:41 dist-test-slave-jt0t krb5kdc[10640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:41.226804 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:41.306007 10653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:41.306221 10653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:41.306254 10653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:41.307402 10653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:41.307442 10653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:41.307461 10653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:41.307476 10653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:41.307493 10653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:41.309005 10653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:41.309240 10653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:41.309446 10653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:41.314075 10659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:41.314051 10658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:41.314085 10661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:41.314142 10653 server_base.cc:1034] running on GCE node
I20260619 02:53:41.314499 10653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:41.314745 10653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:41.315876 10653 hybrid_clock.cc:648] HybridClock initialized: now 1781837621315858 us; error 33 us; skew 500 ppm
Jun 19 02:53:41 dist-test-slave-jt0t krb5kdc[10640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:41.318279 10653 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:41.318717 10653 webserver.cc:460] Webserver started at http://127.0.0.1:43679/ using document root <none> and password file <none>
I20260619 02:53:41.318920 10653 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:41.318964 10653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:41.319075 10653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:41.320124 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "a15d4f78f2a14475a367fbd0cd2aefe3"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-jt0t"
server_key: "58a37d3971c239754031bf8b66161bce"
server_key_iv: "f288039672cc464fabaa55d357aa7b2e"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.320439 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "a15d4f78f2a14475a367fbd0cd2aefe3"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-jt0t"
server_key: "58a37d3971c239754031bf8b66161bce"
server_key_iv: "f288039672cc464fabaa55d357aa7b2e"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.321691 10653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:41.322368 10668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:41.322521 10653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:41.322594 10653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "a15d4f78f2a14475a367fbd0cd2aefe3"
format_stamp: "Formatted at 2026-06-19 02:53:41 on dist-test-slave-jt0t"
server_key: "58a37d3971c239754031bf8b66161bce"
server_key_iv: "f288039672cc464fabaa55d357aa7b2e"
server_key_version: "encryptionkey@0"
I20260619 02:53:41.322660 10653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:41.334201 10653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:41.336107 10653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:41.336248 10653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:41.340124 10653 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35323
I20260619 02:53:41.340159 10720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35323 every 8 connection(s)
I20260619 02:53:41.340481 10653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:41.341394 10721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:41.341408 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 10653
I20260619 02:53:41.341490 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:41.341675 22163 external_mini_cluster.cc:1428] Setting key 728957135be8135f6a1b95a14c3c31e4
I20260619 02:53:41.344740 10721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3: Bootstrap starting.
I20260619 02:53:41.345341 10721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:41.345599 10721 log.cc:824] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:41.346238 10721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3: No bootstrap required, opened a new log
I20260619 02:53:41.347291 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.342177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:46491 (local address 127.0.0.1:35323)
0619 02:53:41.342360 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.342370 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.342404 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:53:41.342883 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.342893 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.342961 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.343112 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:41.343324 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.344674 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.345454 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.345932 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.346042 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.346727 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:41.346731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:41.346732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:41.346886 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:41.346917 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:41.346928 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:41.346995 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.347105 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.347132 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260619 02:53:41.348245 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:41.348284 10721 raft_consensus.cc:348] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d4f78f2a14475a367fbd0cd2aefe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35323 } }
I20260619 02:53:41.348444 10721 raft_consensus.cc:374] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:41.348480 10721 raft_consensus.cc:729] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15d4f78f2a14475a367fbd0cd2aefe3, State: Initialized, Role: FOLLOWER
I20260619 02:53:41.348595 10721 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d4f78f2a14475a367fbd0cd2aefe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35323 } }
I20260619 02:53:41.348678 10721 raft_consensus.cc:388] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:41.348717 10721 raft_consensus.cc:482] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:41.348763 10721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:41.349323 10721 raft_consensus.cc:504] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d4f78f2a14475a367fbd0cd2aefe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35323 } }
I20260619 02:53:41.349419 10721 leader_election.cc:302] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a15d4f78f2a14475a367fbd0cd2aefe3; no voters:
I20260619 02:53:41.349566 10721 leader_election.cc:288] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:41.349632 10730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:41.349764 10730 raft_consensus.cc:686] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [term 1 LEADER]: Becoming Leader. State: Replica: a15d4f78f2a14475a367fbd0cd2aefe3, State: Running, Role: LEADER
I20260619 02:53:41.349896 10721 sys_catalog.cc:564] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:41.349879 10730 consensus_queue.cc:237] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d4f78f2a14475a367fbd0cd2aefe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35323 } }
I20260619 02:53:41.350231 10730 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a15d4f78f2a14475a367fbd0cd2aefe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d4f78f2a14475a367fbd0cd2aefe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35323 } } }
I20260619 02:53:41.350355 10730 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:41.350528 10730 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15d4f78f2a14475a367fbd0cd2aefe3. Latest consensus state: current_term: 1 leader_uuid: "a15d4f78f2a14475a367fbd0cd2aefe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15d4f78f2a14475a367fbd0cd2aefe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35323 } } }
I20260619 02:53:41.350636 10730 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:41.350950 10739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:41.351426 10739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:41.352854 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:41.349583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:36721 (local address 127.0.0.1:35323)
0619 02:53:41.349617 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:41.349619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:41.349768 (+ 149us) server_negotiation.cc:407] Connection header received
0619 02:53:41.349812 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:41.349813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:41.349822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:41.349862 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:41.350037 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.350675 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.351617 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:41.351889 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:41.351949 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:41.352590 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:41.352591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:41.352592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:41.352596 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:41.352610 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:41.352614 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:41.352706 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:41.352777 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 02:53:41.352789 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 02:53:41.353386 10739 catalog_manager.cc:1347] Generated new cluster ID: 06a9ccf806cf49c3ae24aad50e855a82
I20260619 02:53:41.353430 10739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:41.359023 10739 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:41.359517 10739 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:41.367970 10739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a15d4f78f2a14475a367fbd0cd2aefe3: Generated new TSK 0
I20260619 02:53:41.368141 10739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:41.374104 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 10653
W20260619 02:53:41.377820 10638 connection.cc:535] client connection to 127.0.0.1:35323 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35323): Connection reset by peer (error 104)
2026-06-19T02:53:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:41 dist-test-slave-jt0t krb5kdc[10752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:41 dist-test-slave-jt0t krb5kdc[10752](info): set up 2 sockets
Jun 19 02:53:41 dist-test-slave-jt0t krb5kdc[10752](info): commencing operation
krb5kdc: starting...
W20260619 02:53:42.324412 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:42 dist-test-slave-jt0t krb5kdc[10752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:42.439324 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:42.519438 10765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:42.519652 10765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:42.519675 10765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:42.520861 10765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:42.520902 10765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:42.520915 10765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:42.520927 10765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:42.520937 10765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:42.522424 10765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:42.522626 10765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:42.522807 10765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:42.527544 10770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:42.527597 10765 server_base.cc:1034] running on GCE node
W20260619 02:53:42.527514 10771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:42.527525 10773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:42.527978 10765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:42.528251 10765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:42.529425 10765 hybrid_clock.cc:648] HybridClock initialized: now 1781837622529409 us; error 30 us; skew 500 ppm
Jun 19 02:53:42 dist-test-slave-jt0t krb5kdc[10752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:42.531858 10765 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:42.532315 10765 webserver.cc:460] Webserver started at http://127.0.0.1:40035/ using document root <none> and password file <none>
I20260619 02:53:42.532516 10765 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:42.532557 10765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:42.532677 10765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:42.533756 10765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "529b60fc05514cfaad9354aa70cabe33"
format_stamp: "Formatted at 2026-06-19 02:53:42 on dist-test-slave-jt0t"
server_key: "09d637a614407a59aebb49e98eddc91c"
server_key_iv: "8dcfe3dc97dfc943020c2acac36a3e77"
server_key_version: "encryptionkey@0"
I20260619 02:53:42.534058 10765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "529b60fc05514cfaad9354aa70cabe33"
format_stamp: "Formatted at 2026-06-19 02:53:42 on dist-test-slave-jt0t"
server_key: "09d637a614407a59aebb49e98eddc91c"
server_key_iv: "8dcfe3dc97dfc943020c2acac36a3e77"
server_key_version: "encryptionkey@0"
I20260619 02:53:42.535243 10765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:42.535876 10780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:42.536048 10765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:42.536129 10765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "529b60fc05514cfaad9354aa70cabe33"
format_stamp: "Formatted at 2026-06-19 02:53:42 on dist-test-slave-jt0t"
server_key: "09d637a614407a59aebb49e98eddc91c"
server_key_iv: "8dcfe3dc97dfc943020c2acac36a3e77"
server_key_version: "encryptionkey@0"
I20260619 02:53:42.536186 10765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:42.539400 10765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:42.541512 10765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:42.541615 10765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:42.545176 10765 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39729
I20260619 02:53:42.545207 10832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39729 every 8 connection(s)
I20260619 02:53:42.545514 10765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:42.546454 10833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:42.548641 10833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33: Bootstrap starting.
I20260619 02:53:42.549211 10833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:42.549468 10833 log.cc:824] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:42.550222 10833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33: No bootstrap required, opened a new log
I20260619 02:53:42.551776 10833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b60fc05514cfaad9354aa70cabe33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260619 02:53:42.551923 10833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:42.551955 10833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529b60fc05514cfaad9354aa70cabe33, State: Initialized, Role: FOLLOWER
I20260619 02:53:42.552091 10833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b60fc05514cfaad9354aa70cabe33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260619 02:53:42.552167 10833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:42.552199 10833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:42.552248 10833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:42.552829 10833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b60fc05514cfaad9354aa70cabe33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260619 02:53:42.552932 10833 leader_election.cc:302] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 529b60fc05514cfaad9354aa70cabe33; no voters:
I20260619 02:53:42.553087 10833 leader_election.cc:288] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:42.553128 10836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:42.553270 10836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [term 1 LEADER]: Becoming Leader. State: Replica: 529b60fc05514cfaad9354aa70cabe33, State: Running, Role: LEADER
I20260619 02:53:42.553334 10833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:42.553359 10836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b60fc05514cfaad9354aa70cabe33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260619 02:53:42.553714 10838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 529b60fc05514cfaad9354aa70cabe33. Latest consensus state: current_term: 1 leader_uuid: "529b60fc05514cfaad9354aa70cabe33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b60fc05514cfaad9354aa70cabe33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } } }
I20260619 02:53:42.553836 10837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "529b60fc05514cfaad9354aa70cabe33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b60fc05514cfaad9354aa70cabe33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } } }
I20260619 02:53:42.553853 10838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:42.554030 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 10765
I20260619 02:53:42.553920 10837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:42.554139 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:42.554214 10845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:42.554332 22163 external_mini_cluster.cc:1428] Setting key 23fc1d8c3e6a5073849163c3a4f7e336
I20260619 02:53:42.554632 10845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:42.556313 10845 catalog_manager.cc:1347] Generated new cluster ID: 8147f8609c3446ebb8b789bbc256e536
I20260619 02:53:42.556406 10845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:42.557101 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:42.554983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:42711 (local address 127.0.0.1:39729)
0619 02:53:42.555147 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:42.555156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:42.555369 (+ 213us) server_negotiation.cc:407] Connection header received
0619 02:53:42.555522 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:42.555529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:42.555560 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:42.555721 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:42.556644 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:42.556678 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:42.556678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:42.556683 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:42.556725 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:42.556739 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:42.556802 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:42.556892 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 02:53:42.556934 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260619 02:53:42.557940 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:42.560519 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:42.559442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:44333 (local address 127.0.0.1:39729)
0619 02:53:42.559475 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:42.559477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:42.559596 (+ 119us) server_negotiation.cc:407] Connection header received
0619 02:53:42.559654 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:42.559655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:42.559660 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:42.559701 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:42.560298 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:42.560300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:42.560300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:42.560304 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:42.560318 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:42.560323 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:42.560371 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:42.560422 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 02:53:42.560435 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 02:53:42.568630 10845 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:42.568847 10845 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:42.579131 10845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 529b60fc05514cfaad9354aa70cabe33: Generated new TSK 0
I20260619 02:53:42.579334 10845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:42.583262 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 10765
2026-06-19T02:53:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:42 dist-test-slave-jt0t krb5kdc[10864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:42 dist-test-slave-jt0t krb5kdc[10864](info): set up 2 sockets
Jun 19 02:53:42 dist-test-slave-jt0t krb5kdc[10864](info): commencing operation
krb5kdc: starting...
W20260619 02:53:43.540514 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:43 dist-test-slave-jt0t krb5kdc[10864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:43.655529 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:43.734876 10877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:43.735101 10877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:43.735123 10877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:43.736276 10877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:43.736320 10877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:43.736333 10877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:43.736344 10877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:43.736364 10877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:43.737871 10877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45907
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:43.738116 10877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:43.738327 10877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:43.742993 10885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:43.743129 10877 server_base.cc:1034] running on GCE node
W20260619 02:53:43.742966 10882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:43.742957 10883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:43.743456 10877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:43.743745 10877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:43.744915 10877 hybrid_clock.cc:648] HybridClock initialized: now 1781837623744899 us; error 36 us; skew 500 ppm
Jun 19 02:53:43 dist-test-slave-jt0t krb5kdc[10864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:43.747334 10877 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:43.747797 10877 webserver.cc:460] Webserver started at http://127.0.0.1:41863/ using document root <none> and password file <none>
I20260619 02:53:43.748027 10877 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:43.748076 10877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:43.748204 10877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:43.749302 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "9747cedd3fbf40eebdf0487070fd8299"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-jt0t"
server_key: "c29334365de6a018c4101211ff19d9e4"
server_key_iv: "34679591242e22d89402936345a792b2"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.749639 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "9747cedd3fbf40eebdf0487070fd8299"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-jt0t"
server_key: "c29334365de6a018c4101211ff19d9e4"
server_key_iv: "34679591242e22d89402936345a792b2"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.750960 10877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:43.751762 10892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:43.751957 10877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:43.752048 10877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "9747cedd3fbf40eebdf0487070fd8299"
format_stamp: "Formatted at 2026-06-19 02:53:43 on dist-test-slave-jt0t"
server_key: "c29334365de6a018c4101211ff19d9e4"
server_key_iv: "34679591242e22d89402936345a792b2"
server_key_version: "encryptionkey@0"
I20260619 02:53:43.752113 10877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:43.775548 10877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:43.777747 10877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:43.777935 10877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:43.781692 10877 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45907
I20260619 02:53:43.781792 10944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45907 every 8 connection(s)
I20260619 02:53:43.782083 10877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:43.782927 10945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:43.785291 10945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299: Bootstrap starting.
I20260619 02:53:43.785946 10945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:43.786217 10945 log.cc:824] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:43.786895 10945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299: No bootstrap required, opened a new log
I20260619 02:53:43.788466 10945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9747cedd3fbf40eebdf0487070fd8299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } }
I20260619 02:53:43.788611 10945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:43.788632 10945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9747cedd3fbf40eebdf0487070fd8299, State: Initialized, Role: FOLLOWER
I20260619 02:53:43.788820 10945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9747cedd3fbf40eebdf0487070fd8299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } }
I20260619 02:53:43.788884 10945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:43.788906 10945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:43.788952 10945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:43.789495 10945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9747cedd3fbf40eebdf0487070fd8299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } }
I20260619 02:53:43.789587 10945 leader_election.cc:302] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9747cedd3fbf40eebdf0487070fd8299; no voters:
I20260619 02:53:43.789747 10945 leader_election.cc:288] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:43.789784 10948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:43.789922 10948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [term 1 LEADER]: Becoming Leader. State: Replica: 9747cedd3fbf40eebdf0487070fd8299, State: Running, Role: LEADER
I20260619 02:53:43.790011 10945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:43.790051 10948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9747cedd3fbf40eebdf0487070fd8299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } }
I20260619 02:53:43.790552 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 10877
I20260619 02:53:43.790544 10949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9747cedd3fbf40eebdf0487070fd8299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9747cedd3fbf40eebdf0487070fd8299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } } }
I20260619 02:53:43.790613 10949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:43.790669 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:43.790766 10950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9747cedd3fbf40eebdf0487070fd8299. Latest consensus state: current_term: 1 leader_uuid: "9747cedd3fbf40eebdf0487070fd8299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9747cedd3fbf40eebdf0487070fd8299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45907 } } }
I20260619 02:53:43.790807 10950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:43.790844 22163 external_mini_cluster.cc:1428] Setting key e8b91e1c77cc8a32ee3a383bd533f3ce
W20260619 02:53:43.791729 10964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:43.791785 10964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:43.791833 10957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:43.791988 10957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:43.793581 10957 catalog_manager.cc:1347] Generated new cluster ID: 91e2c3ce3903451ab8a20727cf2c4a20
I20260619 02:53:43.793637 10957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:53:43 dist-test-slave-jt0t krb5kdc[10864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 02:53:43.801046 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.791336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:40803 (local address 127.0.0.1:45907)
0619 02:53:43.791573 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.791583 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.791617 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:53:43.791735 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.791741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.791790 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.791971 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:43.792565 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.793843 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.794586 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.795010 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.795103 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.797400 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:43.797404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:43.797404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:43.797418 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:43.799486 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.799842 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.799846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.799846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.799942 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.800149 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.800150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.800150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.800409 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:43.800559 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.800745 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.800845 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":60,"threads_started":1}
I20260619 02:53:43.801949 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:43.806779 10957 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:43.807300 10957 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:43.813027 10957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9747cedd3fbf40eebdf0487070fd8299: Generated new TSK 0
I20260619 02:53:43.813205 10957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:43.813702 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:43.804983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:56901 (local address 127.0.0.1:45907)
0619 02:53:43.805018 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:43.805020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:43.806489 (+ 1469us) server_negotiation.cc:407] Connection header received
0619 02:53:43.806543 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:43.806545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:43.806556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:43.806612 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:43.807273 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.807975 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.809008 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:43.809298 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:43.809359 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:43.810714 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:43.810715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:43.810716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:43.810726 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:43.812224 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.812674 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.812675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.812675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.812738 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:43.812942 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:43.812943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:43.812943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:43.813088 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:43.813189 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:43.813546 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 02:53:43.813612 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:53:43.815685 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 10877
W20260619 02:53:43.819375 10862 connection.cc:535] client connection to 127.0.0.1:45907 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45907): Connection reset by peer (error 104)
2026-06-19T02:53:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:43 dist-test-slave-jt0t krb5kdc[10976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:43 dist-test-slave-jt0t krb5kdc[10976](info): set up 2 sockets
Jun 19 02:53:43 dist-test-slave-jt0t krb5kdc[10976](info): commencing operation
krb5kdc: starting...
W20260619 02:53:44.772442 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:44 dist-test-slave-jt0t krb5kdc[10976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:44.886770 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:44.965246 10989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:44.965469 10989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:44.965497 10989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:44.966643 10989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:44.966683 10989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:44.966696 10989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:44.966707 10989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:44.966718 10989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:44.968145 10989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37707
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:44.968349 10989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:44.968533 10989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:44.973120 10997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:44.973229 10995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:44.973135 10994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:44.973240 10989 server_base.cc:1034] running on GCE node
I20260619 02:53:44.973577 10989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:44.973865 10989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:44.975010 10989 hybrid_clock.cc:648] HybridClock initialized: now 1781837624974991 us; error 35 us; skew 500 ppm
Jun 19 02:53:44 dist-test-slave-jt0t krb5kdc[10976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:44.981137 10989 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:44.981565 10989 webserver.cc:460] Webserver started at http://127.0.0.1:46671/ using document root <none> and password file <none>
I20260619 02:53:44.981752 10989 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:44.981791 10989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:44.981884 10989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:44.982904 10989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "0bb3f497f8054efaa0e8b6e298d8e470"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-jt0t"
server_key: "e4bebc2d9222104732f5a87585d68b52"
server_key_iv: "1d66eaf9e0f5016253fdd892d488326c"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.983206 10989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "0bb3f497f8054efaa0e8b6e298d8e470"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-jt0t"
server_key: "e4bebc2d9222104732f5a87585d68b52"
server_key_iv: "1d66eaf9e0f5016253fdd892d488326c"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.984360 10989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:44.985025 11004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.985183 10989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:44.985263 10989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "0bb3f497f8054efaa0e8b6e298d8e470"
format_stamp: "Formatted at 2026-06-19 02:53:44 on dist-test-slave-jt0t"
server_key: "e4bebc2d9222104732f5a87585d68b52"
server_key_iv: "1d66eaf9e0f5016253fdd892d488326c"
server_key_version: "encryptionkey@0"
I20260619 02:53:44.985320 10989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:44.999848 10989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:45.001668 10989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:45.001775 10989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:45.005421 10989 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37707
I20260619 02:53:45.005532 11056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37707 every 8 connection(s)
I20260619 02:53:45.005795 10989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:45.006645 11057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:45.008922 11057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470: Bootstrap starting.
I20260619 02:53:45.009479 11057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:45.009727 11057 log.cc:824] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:45.010393 11057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470: No bootstrap required, opened a new log
I20260619 02:53:45.011421 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 10989
I20260619 02:53:45.011549 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:45.011730 22163 external_mini_cluster.cc:1428] Setting key ce949607b8083a6d18df825faffca178
I20260619 02:53:45.012266 11057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb3f497f8054efaa0e8b6e298d8e470" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37707 } }
I20260619 02:53:45.012446 11057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:45.012473 11057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb3f497f8054efaa0e8b6e298d8e470, State: Initialized, Role: FOLLOWER
I20260619 02:53:45.012588 11057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb3f497f8054efaa0e8b6e298d8e470" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37707 } }
I20260619 02:53:45.012681 11057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:45.012723 11057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:45.012771 11057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:45.013679 11057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb3f497f8054efaa0e8b6e298d8e470" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37707 } }
I20260619 02:53:45.013767 11057 leader_election.cc:302] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bb3f497f8054efaa0e8b6e298d8e470; no voters:
I20260619 02:53:45.013906 11057 leader_election.cc:288] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:45.013933 11062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:45.014034 11062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [term 1 LEADER]: Becoming Leader. State: Replica: 0bb3f497f8054efaa0e8b6e298d8e470, State: Running, Role: LEADER
I20260619 02:53:45.014163 11062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb3f497f8054efaa0e8b6e298d8e470" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37707 } }
I20260619 02:53:45.014161 11057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:45.014591 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bb3f497f8054efaa0e8b6e298d8e470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb3f497f8054efaa0e8b6e298d8e470" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37707 } } }
I20260619 02:53:45.014668 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:45.014858 11064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bb3f497f8054efaa0e8b6e298d8e470. Latest consensus state: current_term: 1 leader_uuid: "0bb3f497f8054efaa0e8b6e298d8e470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb3f497f8054efaa0e8b6e298d8e470" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37707 } } }
I20260619 02:53:45.015300 11064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:45.015470 11068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:45.016153 11068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:45.017283 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:45.012261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:33635 (local address 127.0.0.1:37707)
0619 02:53:45.012428 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:45.012444 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:45.012478 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:53:45.012601 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:45.012608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:45.012696 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:45.012837 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:45.013034 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:45.013986 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:45.015033 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:45.015644 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:45.015742 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:45.016477 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:45.016480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:45.016481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:45.016715 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:45.016753 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:45.016768 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:45.016848 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:45.017033 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 02:53:45.017060 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260619 02:53:45.017643 11068 catalog_manager.cc:1347] Generated new cluster ID: 21e2089f00134e9482611d405bb61423
I20260619 02:53:45.017691 11068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:45.018167 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:45.021996 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:45.019362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:47425 (local address 127.0.0.1:37707)
0619 02:53:45.019395 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:45.019397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:45.019499 (+ 102us) server_negotiation.cc:407] Connection header received
0619 02:53:45.019538 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:45.019539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:45.019549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:45.019599 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:45.019713 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:45.020187 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:45.020920 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:45.021132 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:45.021184 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:45.021776 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:45.021777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:45.021777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:45.021781 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:45.021796 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:45.021800 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:45.021847 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:45.021912 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 02:53:45.021924 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 02:53:45.026345 11068 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:45.027017 11068 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:45.038250 11068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bb3f497f8054efaa0e8b6e298d8e470: Generated new TSK 0
I20260619 02:53:45.038480 11068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:45.040531 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 10989
W20260619 02:53:45.044307 10974 connection.cc:535] client connection to 127.0.0.1:37707 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37707): Connection reset by peer (error 104)
2026-06-19T02:53:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:45 dist-test-slave-jt0t krb5kdc[11088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:45 dist-test-slave-jt0t krb5kdc[11088](info): set up 2 sockets
Jun 19 02:53:45 dist-test-slave-jt0t krb5kdc[11088](info): commencing operation
krb5kdc: starting...
W20260619 02:53:45.988580 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:46 dist-test-slave-jt0t krb5kdc[11088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 02:53:46.105605 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:46.186975 11101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:46.187206 11101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:46.187239 11101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:46.188439 11101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:46.188484 11101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:46.188503 11101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:46.188519 11101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:46.188534 11101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:46.190186 11101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:46.190455 11101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:46.190670 11101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:46.195273 11107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:46.195312 11101 server_base.cc:1034] running on GCE node
W20260619 02:53:46.195307 11106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:46.195281 11109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:46.195667 11101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:46.195981 11101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:46.197151 11101 hybrid_clock.cc:648] HybridClock initialized: now 1781837626197115 us; error 51 us; skew 500 ppm
Jun 19 02:53:46 dist-test-slave-jt0t krb5kdc[11088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:46.199635 11101 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 02:53:46.200093 11101 webserver.cc:460] Webserver started at http://127.0.0.1:41995/ using document root <none> and password file <none>
I20260619 02:53:46.200316 11101 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:46.200369 11101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:46.200480 11101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:46.201593 11101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "5ea4d68d8bdb4952840746f71e6032ca"
format_stamp: "Formatted at 2026-06-19 02:53:46 on dist-test-slave-jt0t"
server_key: "5560cb93b294698429330068ed123f0b"
server_key_iv: "9ef7e380b3f9f898d4c0f43378fd3670"
server_key_version: "encryptionkey@0"
I20260619 02:53:46.201930 11101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "5ea4d68d8bdb4952840746f71e6032ca"
format_stamp: "Formatted at 2026-06-19 02:53:46 on dist-test-slave-jt0t"
server_key: "5560cb93b294698429330068ed123f0b"
server_key_iv: "9ef7e380b3f9f898d4c0f43378fd3670"
server_key_version: "encryptionkey@0"
I20260619 02:53:46.203219 11101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:46.204000 11116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:46.204174 11101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:46.204252 11101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "5ea4d68d8bdb4952840746f71e6032ca"
format_stamp: "Formatted at 2026-06-19 02:53:46 on dist-test-slave-jt0t"
server_key: "5560cb93b294698429330068ed123f0b"
server_key_iv: "9ef7e380b3f9f898d4c0f43378fd3670"
server_key_version: "encryptionkey@0"
I20260619 02:53:46.204334 11101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:46.207572 11101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:46.209774 11101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:46.209888 11101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:46.213521 11101 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34265
I20260619 02:53:46.213567 11168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34265 every 8 connection(s)
I20260619 02:53:46.213843 11101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:46.214726 11169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:46.217013 11169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca: Bootstrap starting.
I20260619 02:53:46.217551 11169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:46.217804 11169 log.cc:824] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:46.218482 11169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca: No bootstrap required, opened a new log
I20260619 02:53:46.220206 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 11101
I20260619 02:53:46.220161 11169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea4d68d8bdb4952840746f71e6032ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260619 02:53:46.220340 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:46.220355 11169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:46.220388 11169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea4d68d8bdb4952840746f71e6032ca, State: Initialized, Role: FOLLOWER
I20260619 02:53:46.220510 22163 external_mini_cluster.cc:1428] Setting key 7f4ae1b998be43ae03192a42c7381521
I20260619 02:53:46.220541 11169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea4d68d8bdb4952840746f71e6032ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260619 02:53:46.220620 11169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:46.220683 11169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:46.220736 11169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:46.221853 11169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea4d68d8bdb4952840746f71e6032ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260619 02:53:46.221966 11169 leader_election.cc:302] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ea4d68d8bdb4952840746f71e6032ca; no voters:
I20260619 02:53:46.222118 11169 leader_election.cc:288] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:46.222159 11174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:46.222256 11174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [term 1 LEADER]: Becoming Leader. State: Replica: 5ea4d68d8bdb4952840746f71e6032ca, State: Running, Role: LEADER
I20260619 02:53:46.222391 11169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:46.222394 11174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea4d68d8bdb4952840746f71e6032ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } }
I20260619 02:53:46.222710 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:46.220997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:55085 (local address 127.0.0.1:34265)
0619 02:53:46.221234 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:46.221245 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:46.221283 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:53:46.221441 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:46.221448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:46.221479 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:46.221627 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:46.222265 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:46.222269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:46.222269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:46.222275 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:46.222335 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:46.222352 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:46.222391 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:46.222515 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 02:53:46.222561 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":123,"threads_started":1}
I20260619 02:53:46.223417 11175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ea4d68d8bdb4952840746f71e6032ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea4d68d8bdb4952840746f71e6032ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } } }
I20260619 02:53:46.223523 11175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:46.223865 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:46.223760 11176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ea4d68d8bdb4952840746f71e6032ca. Latest consensus state: current_term: 1 leader_uuid: "5ea4d68d8bdb4952840746f71e6032ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea4d68d8bdb4952840746f71e6032ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34265 } } }
I20260619 02:53:46.223907 11189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:46.223976 11176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:46.224388 11189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:46.225924 11189 catalog_manager.cc:1347] Generated new cluster ID: 5570e07d9d9d4f1c8bb0d8743129444a
I20260619 02:53:46.225975 11189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:46.226378 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:46.225171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:57255 (local address 127.0.0.1:34265)
0619 02:53:46.225204 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:46.225206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:46.225446 (+ 240us) server_negotiation.cc:407] Connection header received
0619 02:53:46.225468 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:46.225469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:46.225486 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:46.225541 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:46.226157 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:46.226159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:46.226159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:46.226163 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:46.226177 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:46.226181 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:46.226225 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:46.226301 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 02:53:46.226316 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:53:46.236420 11189 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:46.236603 11189 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:46.241539 11189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ea4d68d8bdb4952840746f71e6032ca: Generated new TSK 0
I20260619 02:53:46.241740 11189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:46.245036 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 11101
2026-06-19T02:53:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:46 dist-test-slave-jt0t krb5kdc[11200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:46 dist-test-slave-jt0t krb5kdc[11200](info): set up 2 sockets
Jun 19 02:53:46 dist-test-slave-jt0t krb5kdc[11200](info): commencing operation
krb5kdc: starting...
W20260619 02:53:47.204515 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:47 dist-test-slave-jt0t krb5kdc[11200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:53:47.315696 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42423
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:47.395351 11213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:47.395584 11213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:47.395617 11213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:47.396839 11213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:47.396888 11213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:47.396906 11213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:47.396924 11213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:47.396938 11213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:47.398499 11213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:42423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:47.398756 11213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:47.398974 11213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:47.404008 11219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:47.403982 11221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:47.404052 11213 server_base.cc:1034] running on GCE node
W20260619 02:53:47.404004 11218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:47.404497 11213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:47.404805 11213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:47.405964 11213 hybrid_clock.cc:648] HybridClock initialized: now 1781837627405953 us; error 30 us; skew 500 ppm
Jun 19 02:53:47 dist-test-slave-jt0t krb5kdc[11200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837627, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:47.408322 11213 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:47.408808 11213 webserver.cc:460] Webserver started at http://127.21.164.254:46223/ using document root <none> and password file <none>
I20260619 02:53:47.409003 11213 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:47.409044 11213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:47.409133 11213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:47.410177 11213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "6e133f727e064686bb1f62a695dc7e08"
format_stamp: "Formatted at 2026-06-19 02:53:47 on dist-test-slave-jt0t"
server_key: "e12298d42021c4254a21614622fd29be"
server_key_iv: "525423469b0e270b8e6156e5392df5b2"
server_key_version: "encryptionkey@0"
I20260619 02:53:47.410462 11213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "6e133f727e064686bb1f62a695dc7e08"
format_stamp: "Formatted at 2026-06-19 02:53:47 on dist-test-slave-jt0t"
server_key: "e12298d42021c4254a21614622fd29be"
server_key_iv: "525423469b0e270b8e6156e5392df5b2"
server_key_version: "encryptionkey@0"
I20260619 02:53:47.411657 11213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:47.412341 11228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:47.412508 11213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:47.412590 11213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "6e133f727e064686bb1f62a695dc7e08"
format_stamp: "Formatted at 2026-06-19 02:53:47 on dist-test-slave-jt0t"
server_key: "e12298d42021c4254a21614622fd29be"
server_key_iv: "525423469b0e270b8e6156e5392df5b2"
server_key_version: "encryptionkey@0"
I20260619 02:53:47.412668 11213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:47.428545 11213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:47.430680 11213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:47.430794 11213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:47.434484 11213 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:42423
I20260619 02:53:47.434515 11280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:42423 every 8 connection(s)
I20260619 02:53:47.434854 11213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:47.435674 11281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:47.438010 11281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08: Bootstrap starting.
I20260619 02:53:47.438547 11281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:47.438774 11281 log.cc:824] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:47.439424 11281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08: No bootstrap required, opened a new log
I20260619 02:53:47.440541 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 11213
I20260619 02:53:47.440725 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:47.440893 22163 external_mini_cluster.cc:1428] Setting key cb08b2fe0a0bee0f600b4b6c08d70394
I20260619 02:53:47.441156 11281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e133f727e064686bb1f62a695dc7e08" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42423 } }
I20260619 02:53:47.441329 11281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:47.441388 11281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e133f727e064686bb1f62a695dc7e08, State: Initialized, Role: FOLLOWER
I20260619 02:53:47.441504 11281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e133f727e064686bb1f62a695dc7e08" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42423 } }
I20260619 02:53:47.441586 11281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:47.441617 11281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:47.441663 11281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:47.442647 11281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e133f727e064686bb1f62a695dc7e08" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42423 } }
I20260619 02:53:47.442752 11281 leader_election.cc:302] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e133f727e064686bb1f62a695dc7e08; no voters:
I20260619 02:53:47.442909 11281 leader_election.cc:288] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:47.442960 11286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:47.443104 11286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [term 1 LEADER]: Becoming Leader. State: Replica: 6e133f727e064686bb1f62a695dc7e08, State: Running, Role: LEADER
I20260619 02:53:47.443152 11281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:47.443192 11286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e133f727e064686bb1f62a695dc7e08" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42423 } }
I20260619 02:53:47.443526 11286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e133f727e064686bb1f62a695dc7e08. Latest consensus state: current_term: 1 leader_uuid: "6e133f727e064686bb1f62a695dc7e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e133f727e064686bb1f62a695dc7e08" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42423 } } }
I20260619 02:53:47.443584 11287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e133f727e064686bb1f62a695dc7e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e133f727e064686bb1f62a695dc7e08" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42423 } } }
I20260619 02:53:47.443653 11286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:47.443661 11287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:47.444362 11294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:47.444909 11294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:47.446568 11294 catalog_manager.cc:1347] Generated new cluster ID: d371d86034a949a2b89adc2ce11e3a5d
I20260619 02:53:47.446614 11294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:53:47 dist-test-slave-jt0t krb5kdc[11200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:47.452127 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:47.441406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60995 (local address 127.21.164.254:42423)
0619 02:53:47.441606 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:47.441616 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:47.441657 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:53:47.441783 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:47.441790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:47.441848 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:47.442005 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:47.442590 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:47.444170 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:47.445212 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:47.445787 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:47.445886 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:47.448548 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:47.448552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:47.448552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:47.448565 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:47.450473 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:47.450906 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:47.450908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:47.450909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:47.451020 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:47.451267 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:47.451268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:47.451268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:47.451551 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:47.451691 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:47.451839 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 02:53:47.451932 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260619 02:53:47.453176 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:47.460287 11294 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:47.460852 11294 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:47.461750 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:47.454662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50537 (local address 127.21.164.254:42423)
0619 02:53:47.454697 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:47.454698 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:47.454705 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:53:47.454724 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:47.454725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:47.454734 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:47.454794 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:47.455350 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:47.456124 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:47.457090 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:47.457441 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:47.457510 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:47.458822 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:47.458823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:47.458824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:47.458831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:47.460376 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:47.460825 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:47.460827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:47.460827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:47.460894 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:47.461123 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:47.461124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:47.461125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:47.461259 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:47.461339 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:47.461591 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 02:53:47.461656 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:53:47.470692 11294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e133f727e064686bb1f62a695dc7e08: Generated new TSK 0
I20260619 02:53:47.470893 11294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:47.477921 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 11213
W20260619 02:53:47.481762 11198 connection.cc:535] client connection to 127.21.164.254:42423 recv error: Network error: failed to read from TLS socket (remote: 127.21.164.254:42423): Connection reset by peer (error 104)
2026-06-19T02:53:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:47 dist-test-slave-jt0t krb5kdc[11312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:47 dist-test-slave-jt0t krb5kdc[11312](info): set up 2 sockets
Jun 19 02:53:47 dist-test-slave-jt0t krb5kdc[11312](info): commencing operation
krb5kdc: starting...
W20260619 02:53:48.432590 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:48 dist-test-slave-jt0t krb5kdc[11312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:53:48.552114 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:41875
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:33151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:41875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:48.631024 11325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:48.631220 11325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:48.631242 11325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:48.632418 11325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:48.632463 11325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:48.632476 11325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:48.632488 11325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:48.632498 11325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:48.634042 11325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:33151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:41875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:41875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:48.634271 11325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:48.634497 11325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:48.639159 11330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:48.639176 11331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:48.639163 11333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:48.639340 11325 server_base.cc:1034] running on GCE node
I20260619 02:53:48.639659 11325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:48.639948 11325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:48.641103 11325 hybrid_clock.cc:648] HybridClock initialized: now 1781837628641083 us; error 38 us; skew 500 ppm
Jun 19 02:53:48 dist-test-slave-jt0t krb5kdc[11312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837628, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:48.643517 11325 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:48.643970 11325 webserver.cc:460] Webserver started at http://127.21.164.254:41553/ using document root <none> and password file <none>
I20260619 02:53:48.644186 11325 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:48.644235 11325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:48.644361 11325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:48.645510 11325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "a9d3e2fbc3774b64915c908ff285ed97"
format_stamp: "Formatted at 2026-06-19 02:53:48 on dist-test-slave-jt0t"
server_key: "ea2d632573778d5cc59c7c4a7d3a6231"
server_key_iv: "b279fa40da18bc434914f5a9e6e216d1"
server_key_version: "encryptionkey@0"
I20260619 02:53:48.645850 11325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "a9d3e2fbc3774b64915c908ff285ed97"
format_stamp: "Formatted at 2026-06-19 02:53:48 on dist-test-slave-jt0t"
server_key: "ea2d632573778d5cc59c7c4a7d3a6231"
server_key_iv: "b279fa40da18bc434914f5a9e6e216d1"
server_key_version: "encryptionkey@0"
I20260619 02:53:48.647209 11325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:53:48.647948 11340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:48.648121 11325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:53:48.648204 11325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "a9d3e2fbc3774b64915c908ff285ed97"
format_stamp: "Formatted at 2026-06-19 02:53:48 on dist-test-slave-jt0t"
server_key: "ea2d632573778d5cc59c7c4a7d3a6231"
server_key_iv: "b279fa40da18bc434914f5a9e6e216d1"
server_key_version: "encryptionkey@0"
I20260619 02:53:48.648286 11325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:48.663587 11325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:48.665609 11325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:48.665748 11325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:48.669643 11325 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:41875
I20260619 02:53:48.669680 11392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:41875 every 8 connection(s)
I20260619 02:53:48.670027 11325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:48.670818 11393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:48.673102 11393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97: Bootstrap starting.
I20260619 02:53:48.673622 11393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:48.673842 11393 log.cc:824] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:48.674470 11393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97: No bootstrap required, opened a new log
I20260619 02:53:48.675987 11393 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d3e2fbc3774b64915c908ff285ed97" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41875 } }
I20260619 02:53:48.676124 11393 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:48.676146 11393 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d3e2fbc3774b64915c908ff285ed97, State: Initialized, Role: FOLLOWER
I20260619 02:53:48.676257 11393 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d3e2fbc3774b64915c908ff285ed97" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41875 } }
I20260619 02:53:48.676333 11393 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:48.676360 11393 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:48.676388 11393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:48.676924 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 11325
I20260619 02:53:48.677050 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:48.677230 22163 external_mini_cluster.cc:1428] Setting key c007490f595da776efb656605710481b
I20260619 02:53:48.677369 11393 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d3e2fbc3774b64915c908ff285ed97" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41875 } }
I20260619 02:53:48.677472 11393 leader_election.cc:302] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9d3e2fbc3774b64915c908ff285ed97; no voters:
I20260619 02:53:48.677665 11396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:48.677812 11396 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [term 1 LEADER]: Becoming Leader. State: Replica: a9d3e2fbc3774b64915c908ff285ed97, State: Running, Role: LEADER
I20260619 02:53:48.677830 11393 leader_election.cc:288] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:48.678128 11393 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:48.678588 11396 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d3e2fbc3774b64915c908ff285ed97" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41875 } }
I20260619 02:53:48.679193 11399 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9d3e2fbc3774b64915c908ff285ed97. Latest consensus state: current_term: 1 leader_uuid: "a9d3e2fbc3774b64915c908ff285ed97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d3e2fbc3774b64915c908ff285ed97" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41875 } } }
I20260619 02:53:48.679294 11399 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:48.679804 11413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:48.679837 11400 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9d3e2fbc3774b64915c908ff285ed97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d3e2fbc3774b64915c908ff285ed97" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 41875 } } }
I20260619 02:53:48.679948 11400 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:48.680357 11413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:48.681921 11413 catalog_manager.cc:1347] Generated new cluster ID: 9cb2d946627a46999e17cebb75d9bf90
I20260619 02:53:48.681972 11413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:48.682691 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:48.677695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38571 (local address 127.21.164.254:41875)
0619 02:53:48.677861 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:48.677871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:48.677907 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:53:48.678048 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:48.678056 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:48.678114 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:48.678249 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:48.678516 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:48.679905 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:48.680800 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:48.681340 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:48.681442 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:48.682113 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:48.682117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:48.682117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:48.682284 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:48.682312 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:48.682323 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:48.682391 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:48.682495 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 02:53:48.682521 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260619 02:53:48.683492 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:48.691244 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:48.688414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39253 (local address 127.21.164.254:41875)
0619 02:53:48.688453 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:48.688455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:48.688581 (+ 126us) server_negotiation.cc:407] Connection header received
0619 02:53:48.688640 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:48.688641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:48.688676 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:48.688722 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:48.688860 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:48.689351 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:48.690088 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:48.690310 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:48.690363 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:48.691004 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:48.691005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:48.691005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:48.691010 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:48.691025 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:48.691029 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:48.691079 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:48.691147 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 02:53:48.691158 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:53:48.692149 11413 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:48.692639 11413 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:48.694810 11413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9d3e2fbc3774b64915c908ff285ed97: Generated new TSK 0
I20260619 02:53:48.694973 11413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:48.697788 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 11325
W20260619 02:53:48.701423 11310 connection.cc:535] client connection to 127.21.164.254:41875 recv error: Network error: failed to read from TLS socket (remote: 127.21.164.254:41875): Connection reset by peer (error 104)
2026-06-19T02:53:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:48 dist-test-slave-jt0t krb5kdc[11424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:48 dist-test-slave-jt0t krb5kdc[11424](info): set up 2 sockets
Jun 19 02:53:48 dist-test-slave-jt0t krb5kdc[11424](info): commencing operation
krb5kdc: starting...
W20260619 02:53:49.648701 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:49 dist-test-slave-jt0t krb5kdc[11424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:53:49.763262 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45599
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:45599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:49.842713 11437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:49.842922 11437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:49.842944 11437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:49.844059 11437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:49.844096 11437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:49.844110 11437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:49.844120 11437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:49.844131 11437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:49.845611 11437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:45599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:49.845825 11437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:49.846014 11437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:49.850816 11445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:49.850863 11442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:49.850829 11443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:49.850879 11437 server_base.cc:1034] running on GCE node
I20260619 02:53:49.851276 11437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:49.851590 11437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:49.852711 11437 hybrid_clock.cc:648] HybridClock initialized: now 1781837629852699 us; error 30 us; skew 500 ppm
Jun 19 02:53:49 dist-test-slave-jt0t krb5kdc[11424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837629, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:49.855119 11437 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:49.855573 11437 webserver.cc:460] Webserver started at http://127.21.164.254:41211/ using document root <none> and password file <none>
I20260619 02:53:49.855808 11437 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:49.855858 11437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:49.855983 11437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:49.857151 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "7e6208a9b48040f99ef209af09c01174"
format_stamp: "Formatted at 2026-06-19 02:53:49 on dist-test-slave-jt0t"
server_key: "c1cd4115aee64b16890934344de42d5e"
server_key_iv: "151c79599611f0f9f39e6dde1181daa8"
server_key_version: "encryptionkey@0"
I20260619 02:53:49.857491 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "7e6208a9b48040f99ef209af09c01174"
format_stamp: "Formatted at 2026-06-19 02:53:49 on dist-test-slave-jt0t"
server_key: "c1cd4115aee64b16890934344de42d5e"
server_key_iv: "151c79599611f0f9f39e6dde1181daa8"
server_key_version: "encryptionkey@0"
I20260619 02:53:49.858767 11437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:49.859493 11452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:49.859676 11437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:49.859768 11437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "7e6208a9b48040f99ef209af09c01174"
format_stamp: "Formatted at 2026-06-19 02:53:49 on dist-test-slave-jt0t"
server_key: "c1cd4115aee64b16890934344de42d5e"
server_key_iv: "151c79599611f0f9f39e6dde1181daa8"
server_key_version: "encryptionkey@0"
I20260619 02:53:49.859840 11437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:49.863145 11437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:49.864966 11437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:49.865091 11437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:49.868824 11437 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45599
I20260619 02:53:49.868880 11504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45599 every 8 connection(s)
I20260619 02:53:49.869185 11437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:49.870131 11505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:49.872277 11505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174: Bootstrap starting.
I20260619 02:53:49.872856 11505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:49.873101 11505 log.cc:824] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:49.873771 11505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174: No bootstrap required, opened a new log
I20260619 02:53:49.875319 11505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6208a9b48040f99ef209af09c01174" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45599 } }
I20260619 02:53:49.875458 11505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:49.875481 11505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6208a9b48040f99ef209af09c01174, State: Initialized, Role: FOLLOWER
I20260619 02:53:49.875605 11505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6208a9b48040f99ef209af09c01174" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45599 } }
I20260619 02:53:49.875665 11505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:49.875685 11505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:49.875731 11505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:49.876727 11505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6208a9b48040f99ef209af09c01174" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45599 } }
I20260619 02:53:49.876837 11505 leader_election.cc:302] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e6208a9b48040f99ef209af09c01174; no voters:
I20260619 02:53:49.876991 11505 leader_election.cc:288] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:49.877079 11508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:49.877223 11508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [term 1 LEADER]: Becoming Leader. State: Replica: 7e6208a9b48040f99ef209af09c01174, State: Running, Role: LEADER
I20260619 02:53:49.877240 11505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:49.877388 11508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6208a9b48040f99ef209af09c01174" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45599 } }
I20260619 02:53:49.877750 11510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6208a9b48040f99ef209af09c01174. Latest consensus state: current_term: 1 leader_uuid: "7e6208a9b48040f99ef209af09c01174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6208a9b48040f99ef209af09c01174" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45599 } } }
I20260619 02:53:49.877774 11509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e6208a9b48040f99ef209af09c01174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6208a9b48040f99ef209af09c01174" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45599 } } }
I20260619 02:53:49.877913 11509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:49.878158 11510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:49.878166 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 11437
I20260619 02:53:49.878266 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:49.878425 22163 external_mini_cluster.cc:1428] Setting key ebe76b3f84cc613ca3231e1e67ce0774
I20260619 02:53:49.878751 11517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:49.879274 11517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:49.880841 11517 catalog_manager.cc:1347] Generated new cluster ID: 491ea6617b5c444b8785995ffae5080d
I20260619 02:53:49.880893 11517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:49.882207 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:49.879047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52951 (local address 127.21.164.254:45599)
0619 02:53:49.880819 (+ 1772us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:49.880828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:49.880863 (+ 35us) server_negotiation.cc:407] Connection header received
0619 02:53:49.880980 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:49.880987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:49.881017 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:49.881155 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:49.881812 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:49.881815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:49.881816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:49.881819 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:49.881855 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:49.881869 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:49.881926 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:49.882003 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 02:53:49.882040 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1718,"thread_start_us":77,"threads_started":1}
I20260619 02:53:49.883077 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:49.885713 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:49.884455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42047 (local address 127.21.164.254:45599)
0619 02:53:49.884485 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:49.884487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:49.884899 (+ 412us) server_negotiation.cc:407] Connection header received
0619 02:53:49.884969 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:49.884969 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:49.884974 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:49.885016 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:49.885533 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:49.885534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:49.885534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:49.885538 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:49.885550 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:49.885554 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:49.885601 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:49.885643 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 02:53:49.885654 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 02:53:49.895378 11517 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:49.895557 11517 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:49.899732 11517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e6208a9b48040f99ef209af09c01174: Generated new TSK 0
I20260619 02:53:49.899945 11517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:49.909484 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 11437
2026-06-19T02:53:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:49 dist-test-slave-jt0t krb5kdc[11536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:49 dist-test-slave-jt0t krb5kdc[11536](info): set up 2 sockets
Jun 19 02:53:49 dist-test-slave-jt0t krb5kdc[11536](info): commencing operation
krb5kdc: starting...
W20260619 02:53:50.864683 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:50 dist-test-slave-jt0t krb5kdc[11536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:53:50.979182 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42543
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:42543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:51.058296 11549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:51.058509 11549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:51.058540 11549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:51.059671 11549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:51.059712 11549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:51.059737 11549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:51.059754 11549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:51.059768 11549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:51.061278 11549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:42543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:42543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:51.061520 11549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:51.061733 11549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:51.066313 11554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:51.066310 11555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:51.066461 11549 server_base.cc:1034] running on GCE node
W20260619 02:53:51.066469 11557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:51.066828 11549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:51.067108 11549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:51.068260 11549 hybrid_clock.cc:648] HybridClock initialized: now 1781837631068240 us; error 35 us; skew 500 ppm
Jun 19 02:53:51 dist-test-slave-jt0t krb5kdc[11536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837631, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:51.074283 11549 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:51.074786 11549 webserver.cc:460] Webserver started at http://127.21.164.254:32991/ using document root <none> and password file <none>
I20260619 02:53:51.074985 11549 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:51.075026 11549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:51.075120 11549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:51.076159 11549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "8aec6e6a57d54e44aa4c24a0e8a8a06b"
format_stamp: "Formatted at 2026-06-19 02:53:51 on dist-test-slave-jt0t"
server_key: "573fc3126c0f429b3098e7184df96c3f"
server_key_iv: "b0446a4381a252279604cc16f9ef5381"
server_key_version: "encryptionkey@0"
I20260619 02:53:51.076464 11549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "8aec6e6a57d54e44aa4c24a0e8a8a06b"
format_stamp: "Formatted at 2026-06-19 02:53:51 on dist-test-slave-jt0t"
server_key: "573fc3126c0f429b3098e7184df96c3f"
server_key_iv: "b0446a4381a252279604cc16f9ef5381"
server_key_version: "encryptionkey@0"
I20260619 02:53:51.077767 11549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:53:51.078444 11564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:51.078622 11549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:51.078708 11549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "8aec6e6a57d54e44aa4c24a0e8a8a06b"
format_stamp: "Formatted at 2026-06-19 02:53:51 on dist-test-slave-jt0t"
server_key: "573fc3126c0f429b3098e7184df96c3f"
server_key_iv: "b0446a4381a252279604cc16f9ef5381"
server_key_version: "encryptionkey@0"
I20260619 02:53:51.078795 11549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:51.091188 11549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:51.092978 11549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:51.093086 11549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:51.096797 11549 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:42543
I20260619 02:53:51.096844 11616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:42543 every 8 connection(s)
I20260619 02:53:51.097151 11549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:51.097950 11617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:51.100119 11617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b: Bootstrap starting.
I20260619 02:53:51.100694 11617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:51.100960 11617 log.cc:824] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:51.101608 11617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b: No bootstrap required, opened a new log
I20260619 02:53:51.103144 11617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec6e6a57d54e44aa4c24a0e8a8a06b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42543 } }
I20260619 02:53:51.103309 11617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:51.103333 11617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aec6e6a57d54e44aa4c24a0e8a8a06b, State: Initialized, Role: FOLLOWER
I20260619 02:53:51.103479 11617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec6e6a57d54e44aa4c24a0e8a8a06b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42543 } }
I20260619 02:53:51.103557 11617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:51.103592 11617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:51.103641 11617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:51.103830 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 11549
I20260619 02:53:51.103947 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:51.104126 22163 external_mini_cluster.cc:1428] Setting key 7d15e938462568b11ab2cd3267d34615
I20260619 02:53:51.104719 11617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec6e6a57d54e44aa4c24a0e8a8a06b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42543 } }
I20260619 02:53:51.104832 11617 leader_election.cc:302] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8aec6e6a57d54e44aa4c24a0e8a8a06b; no voters:
I20260619 02:53:51.104990 11617 leader_election.cc:288] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:51.105011 11622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:51.105109 11622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [term 1 LEADER]: Becoming Leader. State: Replica: 8aec6e6a57d54e44aa4c24a0e8a8a06b, State: Running, Role: LEADER
I20260619 02:53:51.105248 11617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:51.105245 11622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec6e6a57d54e44aa4c24a0e8a8a06b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42543 } }
I20260619 02:53:51.105644 11623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8aec6e6a57d54e44aa4c24a0e8a8a06b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec6e6a57d54e44aa4c24a0e8a8a06b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42543 } } }
I20260619 02:53:51.105652 11624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8aec6e6a57d54e44aa4c24a0e8a8a06b. Latest consensus state: current_term: 1 leader_uuid: "8aec6e6a57d54e44aa4c24a0e8a8a06b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8aec6e6a57d54e44aa4c24a0e8a8a06b" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 42543 } } }
I20260619 02:53:51.105737 11623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:51.105753 11624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:51.106594 11629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:51.107113 11629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:51.108497 11629 catalog_manager.cc:1347] Generated new cluster ID: b6dc13afde1d433884697bfaca60ae11
I20260619 02:53:51.108547 11629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 02:53:51 dist-test-slave-jt0t krb5kdc[11536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:51.114996 11629 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:51.115617 11629 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:51.116082 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:51.104576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36379 (local address 127.21.164.254:42543)
0619 02:53:51.104755 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:51.104765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:51.104795 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:53:51.104915 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:51.104921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:51.104977 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:51.105828 (+ 851us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:51.105938 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:51.107308 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:51.108323 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:51.108921 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:51.109020 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:51.112075 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:51.112080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:51.112080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:51.112097 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:51.114260 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:51.114678 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:51.114682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:51.114682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:51.114787 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:51.115057 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:51.115058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:51.115059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:51.115381 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:51.115543 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:51.115681 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 02:53:51.115844 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260619 02:53:51.117177 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:51.120347 11629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8aec6e6a57d54e44aa4c24a0e8a8a06b: Generated new TSK 0
I20260619 02:53:51.120524 11629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:51.126144 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:51.118654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34145 (local address 127.21.164.254:42543)
0619 02:53:51.118688 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:51.118689 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:51.120070 (+ 1381us) server_negotiation.cc:407] Connection header received
0619 02:53:51.120112 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:51.120113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:51.120124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:51.120168 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:51.121113 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:51.121610 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:51.122331 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:51.122570 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:51.122624 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:51.123818 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:51.123819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:51.123820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:51.123827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:51.125095 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:51.125408 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:51.125409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:51.125409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:51.125474 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:51.125646 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:51.125647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:51.125648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:51.125754 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:51.125845 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:51.125975 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 02:53:51.126056 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 02:53:51.128293 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 11549
2026-06-19T02:53:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:51 dist-test-slave-jt0t krb5kdc[11648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:51 dist-test-slave-jt0t krb5kdc[11648](info): set up 2 sockets
Jun 19 02:53:51 dist-test-slave-jt0t krb5kdc[11648](info): commencing operation
krb5kdc: starting...
W20260619 02:53:52.080550 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:52 dist-test-slave-jt0t krb5kdc[11648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:53:52.195367 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45005
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:45005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:52.275734 11661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:52.275962 11661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:52.275985 11661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:52.277207 11661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:52.277249 11661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:52.277262 11661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:52.277274 11661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:52.277285 11661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:52.278813 11661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:45005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:45005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:52.279054 11661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:52.279277 11661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:52.284070 11666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:52.284097 11661 server_base.cc:1034] running on GCE node
W20260619 02:53:52.284062 11667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:52.284062 11669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:52.284492 11661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:52.284816 11661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:52.285959 11661 hybrid_clock.cc:648] HybridClock initialized: now 1781837632285947 us; error 27 us; skew 500 ppm
Jun 19 02:53:52 dist-test-slave-jt0t krb5kdc[11648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837632, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:52.288250 11661 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:52.288640 11661 webserver.cc:460] Webserver started at http://127.21.164.254:36101/ using document root <none> and password file <none>
I20260619 02:53:52.288892 11661 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:52.288939 11661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:52.289054 11661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:52.290110 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "a671965121f0438daeee835aef7734d9"
format_stamp: "Formatted at 2026-06-19 02:53:52 on dist-test-slave-jt0t"
server_key: "42362d54e21f945c5590f84911d72b92"
server_key_iv: "e699f1e23ad9b4d8fa521d5a8fb7174c"
server_key_version: "encryptionkey@0"
I20260619 02:53:52.290434 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "a671965121f0438daeee835aef7734d9"
format_stamp: "Formatted at 2026-06-19 02:53:52 on dist-test-slave-jt0t"
server_key: "42362d54e21f945c5590f84911d72b92"
server_key_iv: "e699f1e23ad9b4d8fa521d5a8fb7174c"
server_key_version: "encryptionkey@0"
I20260619 02:53:52.291653 11661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 02:53:52.292373 11676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:52.292541 11661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:53:52.292627 11661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "a671965121f0438daeee835aef7734d9"
format_stamp: "Formatted at 2026-06-19 02:53:52 on dist-test-slave-jt0t"
server_key: "42362d54e21f945c5590f84911d72b92"
server_key_iv: "e699f1e23ad9b4d8fa521d5a8fb7174c"
server_key_version: "encryptionkey@0"
I20260619 02:53:52.292737 11661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:52.304229 11661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:52.306063 11661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:52.306195 11661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:52.309903 11661 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:45005
I20260619 02:53:52.309957 11728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:45005 every 8 connection(s)
I20260619 02:53:52.310256 11661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:52.311084 11729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:52.313298 11729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9: Bootstrap starting.
I20260619 02:53:52.313865 11729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:52.314116 11729 log.cc:824] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:52.314818 11729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9: No bootstrap required, opened a new log
I20260619 02:53:52.316349 11729 raft_consensus.cc:348] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a671965121f0438daeee835aef7734d9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45005 } }
I20260619 02:53:52.316490 11729 raft_consensus.cc:374] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:52.316510 11729 raft_consensus.cc:729] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a671965121f0438daeee835aef7734d9, State: Initialized, Role: FOLLOWER
I20260619 02:53:52.316642 11729 consensus_queue.cc:260] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a671965121f0438daeee835aef7734d9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45005 } }
I20260619 02:53:52.316743 11729 raft_consensus.cc:388] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:52.316766 11729 raft_consensus.cc:482] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:52.316810 11729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:52.317633 11729 raft_consensus.cc:504] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a671965121f0438daeee835aef7734d9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45005 } }
I20260619 02:53:52.317719 11729 leader_election.cc:302] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a671965121f0438daeee835aef7734d9; no voters:
I20260619 02:53:52.317875 11729 leader_election.cc:288] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:52.317919 11732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:52.318056 11732 raft_consensus.cc:686] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [term 1 LEADER]: Becoming Leader. State: Replica: a671965121f0438daeee835aef7734d9, State: Running, Role: LEADER
I20260619 02:53:52.318122 11729 sys_catalog.cc:564] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:52.318147 11732 consensus_queue.cc:237] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a671965121f0438daeee835aef7734d9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45005 } }
I20260619 02:53:52.318498 11733 sys_catalog.cc:455] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a671965121f0438daeee835aef7734d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a671965121f0438daeee835aef7734d9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45005 } } }
I20260619 02:53:52.318503 11734 sys_catalog.cc:455] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a671965121f0438daeee835aef7734d9. Latest consensus state: current_term: 1 leader_uuid: "a671965121f0438daeee835aef7734d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a671965121f0438daeee835aef7734d9" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 45005 } } }
I20260619 02:53:52.318624 11733 sys_catalog.cc:458] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:52.318631 11734 sys_catalog.cc:458] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:52.318939 11741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:52.319449 11741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:52.320147 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 11661
I20260619 02:53:52.320257 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:52.320417 22163 external_mini_cluster.cc:1428] Setting key 681c077ec835be767fbad2633bfd01b8
I20260619 02:53:52.330467 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:52.320875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59349 (local address 127.21.164.254:45005)
0619 02:53:52.325021 (+ 4146us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:52.325032 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:52.325069 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:53:52.325200 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:52.325207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:52.325264 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:52.325613 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:52.325709 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:52.327084 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:52.328185 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:52.328859 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:52.328975 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:52.329735 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:52.329740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:52.329741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:52.329953 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:52.329994 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:52.330009 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:52.330071 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:52.330157 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 02:53:52.330223 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4081,"thread_start_us":64,"threads_started":1}
I20260619 02:53:52.330852 11741 catalog_manager.cc:1347] Generated new cluster ID: 57829d443f374a06a6a324b0d21c4608
I20260619 02:53:52.330909 11741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:52.331362 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:52.338807 11741 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:52.339524 11741 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:52.339769 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:52.332697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43073 (local address 127.21.164.254:45005)
0619 02:53:52.332733 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:52.332734 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:52.336907 (+ 4173us) server_negotiation.cc:407] Connection header received
0619 02:53:52.337012 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:52.337013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:52.337026 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:52.337074 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:52.337206 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:52.337713 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:52.338494 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:52.338724 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:52.338782 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:52.339457 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:52.339458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:52.339458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:52.339463 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:52.339480 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:52.339484 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:52.339528 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:52.339639 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 02:53:52.339686 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:53:52.343091 11741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a671965121f0438daeee835aef7734d9: Generated new TSK 0
I20260619 02:53:52.343292 11741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:52.347875 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 11661
2026-06-19T02:53:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:52 dist-test-slave-jt0t krb5kdc[11760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:52 dist-test-slave-jt0t krb5kdc[11760](info): set up 2 sockets
Jun 19 02:53:52 dist-test-slave-jt0t krb5kdc[11760](info): commencing operation
krb5kdc: starting...
W20260619 02:53:53.300519 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:53 dist-test-slave-jt0t krb5kdc[11760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:53:53.417950 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:38935
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:38935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:53.496891 11773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:53.497097 11773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:53.497126 11773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:53.498286 11773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:53.498332 11773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:53.498349 11773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:53.498365 11773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:53.498380 11773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:53.499943 11773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:38935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:38935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:53.500204 11773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:53.500415 11773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:53.505014 11779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:53.505007 11781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:53.505007 11778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:53.505858 11773 server_base.cc:1034] running on GCE node
I20260619 02:53:53.506114 11773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:53.506544 11773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:53.507710 11773 hybrid_clock.cc:648] HybridClock initialized: now 1781837633507687 us; error 35 us; skew 500 ppm
Jun 19 02:53:53 dist-test-slave-jt0t krb5kdc[11760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:53.510943 11773 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:53.511483 11773 webserver.cc:460] Webserver started at http://127.21.164.254:33705/ using document root <none> and password file <none>
I20260619 02:53:53.511754 11773 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:53.511819 11773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:53.511950 11773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:53.513217 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "261392f2471c4642a98d8eb7ebc6bc84"
format_stamp: "Formatted at 2026-06-19 02:53:53 on dist-test-slave-jt0t"
server_key: "6efac2d691f7d1ee0283b66c12ccc7d4"
server_key_iv: "f6c8965f454a0257c367507f1438c7b3"
server_key_version: "encryptionkey@0"
I20260619 02:53:53.513520 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "261392f2471c4642a98d8eb7ebc6bc84"
format_stamp: "Formatted at 2026-06-19 02:53:53 on dist-test-slave-jt0t"
server_key: "6efac2d691f7d1ee0283b66c12ccc7d4"
server_key_iv: "f6c8965f454a0257c367507f1438c7b3"
server_key_version: "encryptionkey@0"
I20260619 02:53:53.514695 11773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:53.515334 11788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:53.515462 11773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:53.515530 11773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "261392f2471c4642a98d8eb7ebc6bc84"
format_stamp: "Formatted at 2026-06-19 02:53:53 on dist-test-slave-jt0t"
server_key: "6efac2d691f7d1ee0283b66c12ccc7d4"
server_key_iv: "f6c8965f454a0257c367507f1438c7b3"
server_key_version: "encryptionkey@0"
I20260619 02:53:53.515580 11773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:53.518821 11773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:53.520913 11773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:53.521019 11773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:53.524528 11773 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:38935
I20260619 02:53:53.524571 11840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:38935 every 8 connection(s)
I20260619 02:53:53.524915 11773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:53.525851 11841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:53.528051 11841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84: Bootstrap starting.
I20260619 02:53:53.528594 11841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:53.528890 11841 log.cc:824] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:53.529605 11841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84: No bootstrap required, opened a new log
I20260619 02:53:53.531170 11841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261392f2471c4642a98d8eb7ebc6bc84" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 38935 } }
I20260619 02:53:53.531312 11841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:53:53.531342 11841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261392f2471c4642a98d8eb7ebc6bc84, State: Initialized, Role: FOLLOWER
I20260619 02:53:53.531486 11841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261392f2471c4642a98d8eb7ebc6bc84" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 38935 } }
I20260619 02:53:53.531559 11841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:53:53.531584 11841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:53:53.531631 11841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:53.532482 11841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261392f2471c4642a98d8eb7ebc6bc84" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 38935 } }
I20260619 02:53:53.532522 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 11773
I20260619 02:53:53.532572 11841 leader_election.cc:302] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261392f2471c4642a98d8eb7ebc6bc84; no voters:
I20260619 02:53:53.532619 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:53.532752 11841 leader_election.cc:288] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:53:53.532785 11844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:53.532861 22163 external_mini_cluster.cc:1428] Setting key 44d0e8fcbbddfbc428a99c4638e6edfe
I20260619 02:53:53.532967 11844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [term 1 LEADER]: Becoming Leader. State: Replica: 261392f2471c4642a98d8eb7ebc6bc84, State: Running, Role: LEADER
I20260619 02:53:53.533002 11841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:53.533047 11844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261392f2471c4642a98d8eb7ebc6bc84" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 38935 } }
I20260619 02:53:53.533485 11845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261392f2471c4642a98d8eb7ebc6bc84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261392f2471c4642a98d8eb7ebc6bc84" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 38935 } } }
I20260619 02:53:53.533607 11845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:53.533809 11846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261392f2471c4642a98d8eb7ebc6bc84. Latest consensus state: current_term: 1 leader_uuid: "261392f2471c4642a98d8eb7ebc6bc84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261392f2471c4642a98d8eb7ebc6bc84" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 38935 } } }
I20260619 02:53:53.533915 11846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:53.534169 11854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:53.534674 11854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:53.535585 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:53.533335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40417 (local address 127.21.164.254:38935)
0619 02:53:53.533476 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:53.533486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:53.533653 (+ 167us) server_negotiation.cc:407] Connection header received
0619 02:53:53.533804 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:53.533812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:53.533844 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:53.533982 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:53.535148 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:53.535151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:53.535152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:53.535156 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:53.535195 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:53.535211 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:53.535278 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:53.535366 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 02:53:53.535409 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":58,"threads_started":1}
I20260619 02:53:53.536202 11854 catalog_manager.cc:1347] Generated new cluster ID: cee24e72d5804d17851bbf40054edc63
I20260619 02:53:53.536253 11854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:53.536610 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:53.538785 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:53.537884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40099 (local address 127.21.164.254:38935)
0619 02:53:53.537918 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:53.537920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:53.537983 (+ 63us) server_negotiation.cc:407] Connection header received
0619 02:53:53.538047 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:53.538047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:53.538052 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:53.538096 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:53.538587 (+ 491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:53.538588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:53.538589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:53:53.538593 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:53.538606 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:53:53.538610 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:53.538655 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:53.538702 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 02:53:53.538715 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:53:53.561890 11854 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:53.562094 11854 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:53.578559 11854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261392f2471c4642a98d8eb7ebc6bc84: Generated new TSK 0
I20260619 02:53:53.578787 11854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:53:53.582923 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 11773
2026-06-19T02:53:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1234 ms)
[----------] 18 tests from AuthTokenIssuingTest (22048 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:53 dist-test-slave-jt0t krb5kdc[11872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:53 dist-test-slave-jt0t krb5kdc[11872](info): set up 2 sockets
Jun 19 02:53:53 dist-test-slave-jt0t krb5kdc[11872](info): commencing operation
krb5kdc: starting...
W20260619 02:53:54.540537 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:54 dist-test-slave-jt0t krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:53:54.656210 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:46165
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:33009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:46165,127.21.164.253:46733,127.21.164.252:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:54.737777 11885 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:53:54.738001 11885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:54.738025 11885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:54.738052 11885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:54.739591 11885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:54.739639 11885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:54.739660 11885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:54.739674 11885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:54.739691 11885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:54.741230 11885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:33009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:46165,127.21.164.253:46733,127.21.164.252:36185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:46165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:54.741473 11885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:54.741672 11885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:54.746237 11893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:54.746322 11890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:54.746390 11885 server_base.cc:1034] running on GCE node
W20260619 02:53:54.746302 11891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:54.746682 11885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:54.746966 11885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:54.748137 11885 hybrid_clock.cc:648] HybridClock initialized: now 1781837634748115 us; error 35 us; skew 500 ppm
Jun 19 02:53:54 dist-test-slave-jt0t krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:54.750774 11885 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:54.751209 11885 webserver.cc:460] Webserver started at http://127.21.164.254:43247/ using document root <none> and password file <none>
I20260619 02:53:54.751411 11885 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:54.751458 11885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:54.751577 11885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:54.752635 11885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "582b19b891ed4b99b100470b6c569d63"
format_stamp: "Formatted at 2026-06-19 02:53:54 on dist-test-slave-jt0t"
server_key: "7235c1c6853d70b2bb884e6f8a152794"
server_key_iv: "a746e5d335f07c49a900c9f5cb31ad23"
server_key_version: "encryptionkey@0"
I20260619 02:53:54.753036 11885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "582b19b891ed4b99b100470b6c569d63"
format_stamp: "Formatted at 2026-06-19 02:53:54 on dist-test-slave-jt0t"
server_key: "7235c1c6853d70b2bb884e6f8a152794"
server_key_iv: "a746e5d335f07c49a900c9f5cb31ad23"
server_key_version: "encryptionkey@0"
I20260619 02:53:54.754271 11885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 02:53:54.754957 11900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:54.755117 11885 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 02:53:54.755199 11885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "582b19b891ed4b99b100470b6c569d63"
format_stamp: "Formatted at 2026-06-19 02:53:54 on dist-test-slave-jt0t"
server_key: "7235c1c6853d70b2bb884e6f8a152794"
server_key_iv: "a746e5d335f07c49a900c9f5cb31ad23"
server_key_version: "encryptionkey@0"
I20260619 02:53:54.755268 11885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:54.775218 11885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:54.777433 11885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:54.777577 11885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:54.781240 11885 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:46165
I20260619 02:53:54.781279 11952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:46165 every 8 connection(s)
I20260619 02:53:54.781602 11885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:54.782480 11953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:54.784030 11953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:53:54 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:54.791025 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 11885
I20260619 02:53:54.791142 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:54.791347 22163 external_mini_cluster.cc:1428] Setting key 581febecaf175a9891a26445a03f0dbe
I20260619 02:53:54.793509 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.784756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.21.164.254:46165)
0619 02:53:54.784907 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:54.784915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:54.785049 (+ 134us) server_negotiation.cc:407] Connection header received
0619 02:53:54.785548 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:54.785550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:54.785566 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:54.785650 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:54.786196 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.786997 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.787850 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.788142 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.788212 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:54.790177 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:54.790184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:54.790184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:54.790190 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:54.791789 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:54.792175 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:54.792178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:54.792178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:54.792281 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:54.792533 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:54.792534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:54.792535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:54.792852 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:54.792928 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:54.793114 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:53:54.793252 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260619 02:53:54.793509 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.784742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:46165 (local address 127.0.0.1:48600)
0619 02:53:54.784929 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:54.784967 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:54.785434 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:54.785703 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:54.785706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:54.785959 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:54.786119 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:54.786123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.787086 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.787087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:54.787781 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:54.787783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.788227 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.788228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:54.788286 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:54.788760 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:54.788769 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:54.790093 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:54.791887 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:54.791894 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:54.791896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:54.792093 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:54.792371 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:54.792372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:54.792372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:54.792465 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:54.792924 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:54.792930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:54.792991 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 02:53:54.793071 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 02:53:54.793311 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":114,"thread_start_us":83,"threads_started":1}
I20260619 02:53:54.794518 11953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:53:54.794871 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.794757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:46733 (local address 127.0.0.1:58802)
0619 02:53:54.794801 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:54.794846 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 02:53:54.795002 11902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.253:46733: connect: Connection refused (error 111)
I20260619 02:53:54.795468 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.795346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:46733 (local address 127.0.0.1:58814)
0619 02:53:54.795414 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:54.795446 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 02:53:54.795639 11953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:46733: Network error: Client connection negotiation failed: client connection to 127.21.164.253:46733: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.21.164.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab.
Entry for principal kudu/127.21.164.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab.
I20260619 02:53:54.816219 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.253:46733
--webserver_interface=127.21.164.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab
--principal=kudu/127.21.164.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:33009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:46165,127.21.164.253:46733,127.21.164.252:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:53:54.844875 11953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } attempt: 1
I20260619 02:53:54.845448 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.845254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:46733 (local address 127.0.0.1:58826)
0619 02:53:54.845354 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:54.845399 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 02:53:54.845774 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.845703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:46733 (local address 127.0.0.1:58842)
0619 02:53:54.845732 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:54.845756 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 02:53:54.845863 11953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:46733: Network error: Client connection negotiation failed: client connection to 127.21.164.253:46733: connect: Connection refused (error 111)
W20260619 02:53:54.898370 11960 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:53:54.898591 11960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:54.898607 11960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:54.898628 11960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:54.899811 11960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:54.899854 11960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:54.899866 11960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:54.899878 11960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:54.899888 11960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:54.901688 11960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:33009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:46165,127.21.164.253:46733,127.21.164.252:36185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab
--principal=kudu/127.21.164.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.253:46733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.164.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:54.902058 11960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:54.902275 11960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:53:54.907126 11960 server_base.cc:1034] running on GCE node
W20260619 02:53:54.907084 11965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:54.907075 11968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:54.907083 11966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:54.907615 11960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:54.907898 11960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:54.909034 11960 hybrid_clock.cc:648] HybridClock initialized: now 1781837634909022 us; error 30 us; skew 500 ppm
Jun 19 02:53:54 dist-test-slave-jt0t krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:54.911432 11960 init.cc:377] Logged in from keytab as kudu/127.21.164.253@KRBTEST.COM (short username kudu)
I20260619 02:53:54.911903 11960 webserver.cc:460] Webserver started at http://127.21.164.253:38223/ using document root <none> and password file <none>
I20260619 02:53:54.912106 11960 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:54.912153 11960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:54.912276 11960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:54.913358 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/data/instance:
uuid: "af5502c0bb1c4faf9ee421dbb4686bdc"
format_stamp: "Formatted at 2026-06-19 02:53:54 on dist-test-slave-jt0t"
server_key: "02051bc0861115f00b6898b3d26c2df4"
server_key_iv: "ab52a2f9567f09c03e6b6919a3866419"
server_key_version: "encryptionkey@0"
I20260619 02:53:54.913695 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/wal/instance:
uuid: "af5502c0bb1c4faf9ee421dbb4686bdc"
format_stamp: "Formatted at 2026-06-19 02:53:54 on dist-test-slave-jt0t"
server_key: "02051bc0861115f00b6898b3d26c2df4"
server_key_iv: "ab52a2f9567f09c03e6b6919a3866419"
server_key_version: "encryptionkey@0"
I20260619 02:53:54.914018 11953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } attempt: 2
I20260619 02:53:54.914366 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.914254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:46733 (local address 127.0.0.1:58858)
0619 02:53:54.914303 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:54.914339 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 02:53:54.914695 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.914570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:46733 (local address 127.0.0.1:58862)
0619 02:53:54.914627 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:54.914662 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:46733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 02:53:54.914790 11953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:46733: Network error: Client connection negotiation failed: client connection to 127.21.164.253:46733: connect: Connection refused (error 111)
I20260619 02:53:54.915295 11960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:54.915940 11975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:54.916077 11960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:54.916152 11960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/wal
uuid: "af5502c0bb1c4faf9ee421dbb4686bdc"
format_stamp: "Formatted at 2026-06-19 02:53:54 on dist-test-slave-jt0t"
server_key: "02051bc0861115f00b6898b3d26c2df4"
server_key_iv: "ab52a2f9567f09c03e6b6919a3866419"
server_key_version: "encryptionkey@0"
I20260619 02:53:54.916218 11960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:54.935913 11960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:54.938159 11960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:54.938292 11960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:54.942076 11960 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.253:46733
I20260619 02:53:54.942112 12027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.253:46733 every 8 connection(s)
I20260619 02:53:54.942440 11960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
I20260619 02:53:54.943298 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:54.944878 12028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:53:54 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:54.950991 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 11960
I20260619 02:53:54.951105 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-1/wal/instance
I20260619 02:53:54.951273 22163 external_mini_cluster.cc:1428] Setting key 282f31eaac3b3fda2142b299f84607de
I20260619 02:53:54.953598 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.945633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.21.164.254:46165)
0619 02:53:54.945708 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:54.945711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:54.945897 (+ 186us) server_negotiation.cc:407] Connection header received
0619 02:53:54.946379 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:54.946380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:54.946395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:54.946444 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:54.947047 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.947537 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.948481 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.948732 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.948789 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:54.950814 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:54.950816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:54.950816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:54.950832 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:54.952201 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:54.952581 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:54.952582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:54.952583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:54.952637 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:54.952954 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:54.952954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:54.952954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:54.953062 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:54.953106 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:54.953456 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 02:53:54.953532 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 02:53:54.953773 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.945561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:46165 (local address 127.0.0.1:48604)
0619 02:53:54.945771 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:54.945807 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:54.946281 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:54.946518 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:54.946522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:54.946762 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:54.946964 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:54.946968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.947620 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.947621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:54.948402 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:54.948403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.948803 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.948804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:54.948922 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:54.949398 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:54.949406 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:54.950731 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:54.952281 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:54.952287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:54.952290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:54.952515 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:54.952731 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:54.952733 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:54.952733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:54.952885 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:54.953117 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:54.953123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:54.953292 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 02:53:54.953409 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 02:53:54.953563 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":104,"threads_started":1}
I20260619 02:53:54.954412 12028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.164.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.252@KRBTEST.COM" created.
Jun 19 02:53:54 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
I20260619 02:53:54.963482 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.954662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:46733 (local address 127.0.0.1:58868)
0619 02:53:54.954704 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:54.954714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:54.954749 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:54.954985 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:54.954986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:54.955118 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:54.955162 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:54.955163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.955975 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.955976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:54.956466 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:54.956467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.956955 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.956956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:54.957015 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:54.957414 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:54.957419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:54.960782 (+ 3363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:54.962472 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:54.962474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:54.962474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:54.962689 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:54.962868 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:54.962869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:54.962869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:54.962920 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:54.963172 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:54.963172 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:54.963208 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:53:54.963267 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:53:54.963366 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 02:53:54.963544 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.954735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.21.164.253:46733)
0619 02:53:54.954874 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:54.954878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:54.954893 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:53:54.954908 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:54.954910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:54.954920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:54.954977 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:54.955222 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.955898 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.956531 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:54.956894 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:54.956953 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:54.960888 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:54.960897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:54.960898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:54.960905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:54.962389 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:54.962760 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:54.962763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:54.962763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:54.962815 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:54.962970 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:54.962971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:54.962971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:54.963114 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:54.963187 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:54.963302 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 02:53:54.963417 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260619 02:53:54.964383 12028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:53:54.964746 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.964605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36592)
0619 02:53:54.964643 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:54.964707 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 02:53:54.964882 11977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
I20260619 02:53:54.965246 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:54.965160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36598)
0619 02:53:54.965200 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:54.965223 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 02:53:54.965344 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:36185: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab.
Entry for principal kudu/127.21.164.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab.
I20260619 02:53:54.976099 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.252:36185
--webserver_interface=127.21.164.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab
--principal=kudu/127.21.164.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:33009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:46165,127.21.164.253:46733,127.21.164.252:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:53:55.005954 11953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } attempt: 3
Jun 19 02:53:55 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
I20260619 02:53:55.013370 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.006360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:46733 (local address 127.0.0.1:58880)
0619 02:53:55.006431 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.006445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:55.006503 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:55.006635 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:55.006636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:55.006778 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:55.006850 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.006851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.007777 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.007778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.008320 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.008321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.008863 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.008864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.008920 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.009363 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:55.009367 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:55.010530 (+ 1163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:55.012282 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.012284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.012284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.012455 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.012712 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.012713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.012713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.012786 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.013089 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:55.013090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:55.013136 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:53:55.013195 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 02:53:55.013294 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:53:55.013409 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.006419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.21.164.253:46733)
0619 02:53:55.006489 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.006491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.006503 (+ 12us) server_negotiation.cc:407] Connection header received
0619 02:53:55.006554 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.006555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.006566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.006625 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.006927 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.007686 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.008400 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.008796 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.008866 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.010626 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.010627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.010627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.010636 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.012160 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.012531 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.012532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.012532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.012603 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.012853 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.012854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.012855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.013005 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.013080 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.013233 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.013316 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:53:55.014092 11953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:53:55.014398 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.014307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36606)
0619 02:53:55.014351 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.014376 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 02:53:55.014550 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } attempt: 1
I20260619 02:53:55.014716 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.014618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36622)
0619 02:53:55.014658 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.014690 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 02:53:55.014832 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.014745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36638)
0619 02:53:55.014786 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.014807 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 02:53:55.014823 11953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:36185: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
I20260619 02:53:55.015071 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.014994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36654)
0619 02:53:55.015028 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.015051 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 02:53:55.015147 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:36185: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
I20260619 02:53:55.045987 11953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } attempt: 1
I20260619 02:53:55.046482 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.046330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36662)
0619 02:53:55.046397 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.046443 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 02:53:55.046921 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.046745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36664)
0619 02:53:55.046859 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.046891 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260619 02:53:55.047068 11953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:36185: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
W20260619 02:53:55.058568 12035 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:53:55.058787 12035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:55.058804 12035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:55.058826 12035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:55.060032 12035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:55.060074 12035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:55.060087 12035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:55.060098 12035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:55.060108 12035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:55.061657 12035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:33009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:46165,127.21.164.253:46733,127.21.164.252:36185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab
--principal=kudu/127.21.164.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.252:36185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.164.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:55.061901 12035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:55.062111 12035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:55.066838 12040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:55.066840 12043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:55.066891 12041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:55.067013 12035 server_base.cc:1034] running on GCE node
I20260619 02:53:55.067317 12035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:55.067588 12035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:55.068725 12035 hybrid_clock.cc:648] HybridClock initialized: now 1781837635068706 us; error 27 us; skew 500 ppm
Jun 19 02:53:55 dist-test-slave-jt0t krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837635, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:55.071065 12035 init.cc:377] Logged in from keytab as kudu/127.21.164.252@KRBTEST.COM (short username kudu)
I20260619 02:53:55.071548 12035 webserver.cc:460] Webserver started at http://127.21.164.252:38817/ using document root <none> and password file <none>
I20260619 02:53:55.071751 12035 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:55.071801 12035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:55.071925 12035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:55.073045 12035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/data/instance:
uuid: "c2d8b15a51124d01bcd57348421b8d83"
format_stamp: "Formatted at 2026-06-19 02:53:55 on dist-test-slave-jt0t"
server_key: "9dbf732a655caba1bef29449b2429d01"
server_key_iv: "8a215044bd764a8c55a010f3e6e48284"
server_key_version: "encryptionkey@0"
I20260619 02:53:55.073385 12035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/wal/instance:
uuid: "c2d8b15a51124d01bcd57348421b8d83"
format_stamp: "Formatted at 2026-06-19 02:53:55 on dist-test-slave-jt0t"
server_key: "9dbf732a655caba1bef29449b2429d01"
server_key_iv: "8a215044bd764a8c55a010f3e6e48284"
server_key_version: "encryptionkey@0"
I20260619 02:53:55.074640 12035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:55.075417 12050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:55.075579 12035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:53:55.075657 12035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/wal
uuid: "c2d8b15a51124d01bcd57348421b8d83"
format_stamp: "Formatted at 2026-06-19 02:53:55 on dist-test-slave-jt0t"
server_key: "9dbf732a655caba1bef29449b2429d01"
server_key_iv: "8a215044bd764a8c55a010f3e6e48284"
server_key_version: "encryptionkey@0"
I20260619 02:53:55.075718 12035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:55.083298 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } attempt: 2
I20260619 02:53:55.083734 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.083617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36676)
0619 02:53:55.083669 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.083703 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 02:53:55.084067 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.083985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36688)
0619 02:53:55.084025 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.084049 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:53:55.084142 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:36185: Network error: Client connection negotiation failed: client connection to 127.21.164.252:36185: connect: Connection refused (error 111)
I20260619 02:53:55.095275 12035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:55.097421 12035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:55.097540 12035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:55.101284 12035 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.252:36185
I20260619 02:53:55.101377 12102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.252:36185 every 8 connection(s)
I20260619 02:53:55.101676 12035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
I20260619 02:53:55.102587 12103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:55.104139 12103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:53:55 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837635, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:55.110984 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 12035
I20260619 02:53:55.111093 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837375929108-22163-0/minicluster-data/master-2/wal/instance
I20260619 02:53:55.111279 22163 external_mini_cluster.cc:1428] Setting key b79559004f76818b94d8be639868b72b
I20260619 02:53:55.112600 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.104885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48618 (local address 127.21.164.254:46165)
0619 02:53:55.104950 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.104952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.105113 (+ 161us) server_negotiation.cc:407] Connection header received
0619 02:53:55.105595 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.105596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.105609 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.105661 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.106274 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.106750 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.107708 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.107905 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.107960 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.109978 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.109980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.109980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.109988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.111230 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.111675 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.111677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.111677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.111740 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.111996 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.111997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.111998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.112112 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.112169 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.112428 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.112513 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:53:55.112789 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.104848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:46165 (local address 127.0.0.1:48618)
0619 02:53:55.104998 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.105034 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:55.105497 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:55.105726 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:55.105729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:55.105987 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:55.106184 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.106188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.106853 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.106854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.107624 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.107625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.107972 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.107973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.108091 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.108557 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:55.108565 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:55.109894 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:55.111304 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.111311 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.111313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.111610 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.111841 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.111842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.111842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.111935 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.112168 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:55.112175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:55.112310 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 02:53:55.112397 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 02:53:55.112552 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260619 02:53:55.113282 12103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:53:55 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
Jun 19 02:53:55 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837635, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
I20260619 02:53:55.119647 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.111657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.21.164.252:36185)
0619 02:53:55.111831 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.111836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.111861 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:53:55.111914 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.111916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.111930 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.111993 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.112564 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.113465 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.114186 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.114638 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.114708 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.116939 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.116947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.116948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.116954 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.118326 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.118734 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.118738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.118738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.118798 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.119094 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.119095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.119095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.119249 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.119428 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.119467 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.119544 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260619 02:53:55.119988 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.113521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:46733 (local address 127.0.0.1:58886)
0619 02:53:55.113628 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.113637 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:55.113674 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:55.113774 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:55.113775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:55.113901 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:55.113942 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.113943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.114711 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.114712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.115407 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.115410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.115731 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.115732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.115794 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.116154 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:55.116159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:55.117401 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:55.119000 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.119001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.119001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.119192 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.119377 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.119378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.119378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.119438 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.119708 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:55.119709 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:55.119748 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 02:53:55.119816 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 02:53:55.119901 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 02:53:55.119997 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.113592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.21.164.253:46733)
0619 02:53:55.113634 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.113636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.113663 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:53:55.113719 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.113720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.113729 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.113773 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.114006 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.114638 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.115483 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.115669 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.115718 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.117483 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.117485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.117485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.117492 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.118899 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.119264 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.119265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.119266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.119323 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.119494 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.119495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.119495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.119620 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.119679 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.119850 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.119917 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 02:53:55.120385 12103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:53:55.122328 11953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } attempt: 2
Jun 19 02:53:55 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837635, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
Jun 19 02:53:55 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:55.126708 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.120678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36702)
0619 02:53:55.120709 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.120718 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:55.120756 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:55.120859 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:55.120859 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:55.120977 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:55.121004 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.121005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.121579 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.121580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.122038 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.122039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.122412 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.122413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.122454 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.122885 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:55.122888 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:55.124142 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:55.125728 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.125729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.125729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.125913 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.126082 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.126082 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.126083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.126127 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.126379 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:55.126380 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:55.126421 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 02:53:55.126487 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 02:53:55.126583 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 02:53:55.126772 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.120697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.21.164.252:36185)
0619 02:53:55.120724 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.120726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.120745 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:53:55.120793 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.120794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.120802 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.120851 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.121058 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.121518 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.122098 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.122319 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.122371 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.124218 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.124219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.124219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.124226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.125670 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.125987 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.125987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.125988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.126029 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.126181 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.126182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.126183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.126309 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.126380 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.126521 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.126633 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":15}
Jun 19 02:53:55 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
I20260619 02:53:55.128613 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.120576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.21.164.254:46165)
0619 02:53:55.120619 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.120621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.120630 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:55.120679 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.120680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.120690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.120792 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.121202 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.121649 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.122274 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.122493 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.122547 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.125121 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.125121 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.125122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.125128 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.126639 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.127371 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.127372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.127372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.127425 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.127643 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.127644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.127644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.127995 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.128415 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.128459 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.128529 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:53:55.128701 12103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83: Bootstrap starting.
I20260619 02:53:55.129434 12103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:55.129729 12103 log.cc:824] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:55.129935 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.122657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36710 (local address 127.21.164.252:36185)
0619 02:53:55.123057 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.123059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.123066 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:53:55.123083 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.123084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.123093 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.123161 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.123427 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.123899 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.125236 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.125481 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.125539 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.127501 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.127502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.127503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.127508 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.128845 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.129173 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.129175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.129175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.129233 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.129416 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.129417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.129417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.129570 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.129628 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.129763 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.129825 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":85,"threads_started":1}
I20260619 02:53:55.129938 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.122568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36710)
0619 02:53:55.122608 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.122618 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:55.122648 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:55.123187 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:55.123188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:55.123331 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:55.123361 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.123363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.123965 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.123966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.125158 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.125161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.125800 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.125801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.125871 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.126134 (+ 263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:55.126138 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:55.127406 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:55.128926 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.128927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.128927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.129107 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.129301 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.129302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.129302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.129365 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.129622 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:55.129623 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:55.129675 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 02:53:55.129735 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 02:53:55.129854 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 02:53:55.131028 12103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83: No bootstrap required, opened a new log
I20260619 02:53:55.131049 11953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63: Bootstrap starting.
I20260619 02:53:55.131659 11953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:55.131951 11953 log.cc:824] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:55.132783 11953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63: No bootstrap required, opened a new log
I20260619 02:53:55.132786 12103 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } }
I20260619 02:53:55.132944 12103 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d8b15a51124d01bcd57348421b8d83, State: Initialized, Role: FOLLOWER
I20260619 02:53:55.133030 12103 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } }
I20260619 02:53:55.133312 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } } }
I20260619 02:53:55.133371 12103 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:55.133476 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:55.134490 11953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } }
I20260619 02:53:55.134621 11953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582b19b891ed4b99b100470b6c569d63, State: Initialized, Role: FOLLOWER
I20260619 02:53:55.134696 11953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } }
Jun 19 02:53:55 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
I20260619 02:53:55.135214 11953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:53:55.135619 12122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:55.135715 12122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:55.136497 12123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } } }
I20260619 02:53:55.136600 12123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:55.135509 12035 master_runner.cc:186] Error getting master registration for 127.21.164.254:46165: OK, instance_id { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" instance_seqno: 1781837634780136 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:53:55.137704 12035 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:53:55.138075 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.129991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.21.164.253:46733)
0619 02:53:55.130035 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.130037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.130047 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:53:55.130072 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.130072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.130084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.130137 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.130705 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.131363 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.132330 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.132601 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.132699 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.135226 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.135227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.135228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.135235 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.136846 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.137215 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.137216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.137216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.137272 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.137447 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.137448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.137449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.137570 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.137691 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.137904 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.137971 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:53:55.137580 11885 master_runner.cc:186] Error getting master registration for 127.21.164.253:46733: OK, instance_id { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" instance_seqno: 1781837634941007 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:53:55.140264 11885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 02:53:55.141443 12135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:55.141487 12135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:55.141568 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:53:55.142382 11927 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "582b19b891ed4b99b100470b6c569d63"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48630
I20260619 02:53:55.142481 11927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 02:53:55.142514 11927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:55.143193 11927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } }
I20260619 02:53:55.143402 11927 leader_election.cc:288] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [CANDIDATE]: Term 1 election: Requested vote from peers af5502c0bb1c4faf9ee421dbb4686bdc (127.21.164.253:46733), c2d8b15a51124d01bcd57348421b8d83 (127.21.164.252:36185)
I20260619 02:53:55.143723 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "582b19b891ed4b99b100470b6c569d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c2d8b15a51124d01bcd57348421b8d83"
I20260619 02:53:55.143883 12077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:55.144876 12077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582b19b891ed4b99b100470b6c569d63 in term 1.
I20260619 02:53:55.145110 11902 leader_election.cc:302] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 582b19b891ed4b99b100470b6c569d63, c2d8b15a51124d01bcd57348421b8d83; no voters:
I20260619 02:53:55.145359 12123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:55.145632 12123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [term 1 LEADER]: Becoming Leader. State: Replica: 582b19b891ed4b99b100470b6c569d63, State: Running, Role: LEADER
I20260619 02:53:55.145807 12123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } }
I20260619 02:53:55.146479 12123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 582b19b891ed4b99b100470b6c569d63. Latest consensus state: current_term: 1 leader_uuid: "582b19b891ed4b99b100470b6c569d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } } }
I20260619 02:53:55.146641 12123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:55.148145 12146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:55.148347 12146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:55.150213 12077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [term 1 FOLLOWER]: Refusing update from remote peer 582b19b891ed4b99b100470b6c569d63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:55.150537 12123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:55.151850 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 582b19b891ed4b99b100470b6c569d63. Latest consensus state: current_term: 1 leader_uuid: "582b19b891ed4b99b100470b6c569d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } } }
I20260619 02:53:55.151938 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:55.152428 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.144504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48634 (local address 127.21.164.254:46165)
0619 02:53:55.144546 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.144548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.144748 (+ 200us) server_negotiation.cc:407] Connection header received
0619 02:53:55.144838 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.144839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.144849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.145022 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.145830 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.146496 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.147611 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.147910 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.147973 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.149501 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.149502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.149503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.149510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.151255 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.151605 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.151606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.151606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.151656 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.151859 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.151860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.151860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.151979 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.152072 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.152264 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.152348 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:53:55.153518 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "582b19b891ed4b99b100470b6c569d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } } }
I20260619 02:53:55.153599 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:55.153650 12123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "582b19b891ed4b99b100470b6c569d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } } }
I20260619 02:53:55.153714 12123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:55.158048 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.145527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.21.164.253:46733)
0619 02:53:55.145564 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.145566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.146082 (+ 516us) server_negotiation.cc:407] Connection header received
0619 02:53:55.146174 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.146175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.146186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.146214 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.146926 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.147611 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.149054 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.149345 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.149387 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.155309 (+ 5922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.155310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.155311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.155319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.156811 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.157183 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.157184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.157184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.157243 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.157434 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.157435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.157435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.157562 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.157654 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.157875 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.157957 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 02:53:55.160286 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.145829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.21.164.252:36185)
0619 02:53:55.145865 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.145867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.152499 (+ 6632us) server_negotiation.cc:407] Connection header received
0619 02:53:55.152522 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.152523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.152533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.152572 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.153097 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.153808 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.154690 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.155017 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.155079 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.157505 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.157506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.157507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.157513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.159163 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.159517 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.159519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.159519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.159575 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.159761 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.159762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.159762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.159898 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.159987 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.160106 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.160196 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:53:55.167438 12146 catalog_manager.cc:1347] Generated new cluster ID: a8412454340a4183859f08fa204b3ef8
I20260619 02:53:55.167501 12146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:55.175267 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } attempt: 3
Jun 19 02:53:55 dist-test-slave-jt0t krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837634, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
I20260619 02:53:55.183476 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.175578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:36185 (local address 127.0.0.1:36724)
0619 02:53:55.175677 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:55.175689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:55.175731 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:55.175874 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:55.175874 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:55.176061 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:55.176123 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.176126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.177098 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.177100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.177930 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:55.177933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.178416 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.178417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:55.178477 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.178983 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:55.178989 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:55.180734 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:55.182445 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.182446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.182447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.182639 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.182838 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:55.182839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:55.182839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:55.182887 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:55.183165 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:55.183166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:55.183229 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 02:53:55.183281 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 02:53:55.183376 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 02:53:55.184019 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:55.175675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.21.164.252:36185)
0619 02:53:55.175744 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.175746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.175788 (+ 42us) server_negotiation.cc:407] Connection header received
0619 02:53:55.175811 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.175812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.175822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.175866 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.176215 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.176989 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.178032 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:55.178333 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:55.178658 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:55.180816 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:55.180817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:55.180817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:55.180824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:55.182364 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.182718 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.182720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.182720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.182784 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:55.182952 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:55.182952 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:55.182953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:55.183088 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:55.183135 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:55.183857 (+ 722us) server_negotiation.cc:299] Negotiation successful
0619 02:53:55.183933 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:53:55.185717 12028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc: Bootstrap starting.
I20260619 02:53:55.186470 12028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:55.186770 12028 log.cc:824] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:55.187719 12028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc: No bootstrap required, opened a new log
I20260619 02:53:55.189736 12028 raft_consensus.cc:348] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } }
I20260619 02:53:55.189890 12028 raft_consensus.cc:729] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5502c0bb1c4faf9ee421dbb4686bdc, State: Initialized, Role: FOLLOWER
I20260619 02:53:55.189970 12028 consensus_queue.cc:260] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } }
I20260619 02:53:55.190315 12028 sys_catalog.cc:564] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:55.190404 12001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:55.191210 12001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [term 1 FOLLOWER]: Refusing update from remote peer 582b19b891ed4b99b100470b6c569d63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:55.191387 12002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "582b19b891ed4b99b100470b6c569d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc"
I20260619 02:53:55.191551 12142 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:55.192366 12002 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582b19b891ed4b99b100470b6c569d63 in term 1.
I20260619 02:53:55.190295 12153 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } } }
I20260619 02:53:55.194139 12153 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:55.194265 12153 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 582b19b891ed4b99b100470b6c569d63. Latest consensus state: current_term: 1 leader_uuid: "582b19b891ed4b99b100470b6c569d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } } }
I20260619 02:53:55.194373 12153 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:55.194439 12153 sys_catalog.cc:455] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "582b19b891ed4b99b100470b6c569d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } } }
I20260619 02:53:55.194527 12153 sys_catalog.cc:458] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:55.195218 12146 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:55.195937 12146 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:55.197073 12165 mvcc.cc:205] Tried to move back new op lower bound from 7298406953572265984 to 7298406953558626304. Current Snapshot: MvccSnapshot[applied={T|T < 7298406953572265984}]
I20260619 02:53:55.198455 12123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "582b19b891ed4b99b100470b6c569d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582b19b891ed4b99b100470b6c569d63" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 46165 } } peers { permanent_uuid: "af5502c0bb1c4faf9ee421dbb4686bdc" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 46733 } } peers { permanent_uuid: "c2d8b15a51124d01bcd57348421b8d83" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 36185 } } }
I20260619 02:53:55.198545 12123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63 [sys.catalog]: This master's current role is: LEADER
W20260619 02:53:55.198930 12168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:55.199100 12168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:55.203967 12146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 582b19b891ed4b99b100470b6c569d63: Generated new TSK 0
I20260619 02:53:55.204246 12146 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:53:55.229133 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.227512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.21.164.253:46733)
0619 02:53:55.227556 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.227558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.228441 (+ 883us) server_negotiation.cc:407] Connection header received
0619 02:53:55.228548 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.228550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.228562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.228591 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.229061 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58916: BlockingRecv error: recv got EOF from 127.0.0.1:58916 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 02:53:55.230052 12108 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.227843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.21.164.252:36185)
0619 02:53:55.227882 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.227884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.229547 (+ 1663us) server_negotiation.cc:407] Connection header received
0619 02:53:55.229639 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.229640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.229668 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.229697 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.229988 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36728: BlockingRecv error: recv got EOF from 127.0.0.1:36728 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 02:53:55.230896 11955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.228167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.21.164.254:46165)
0619 02:53:55.228216 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.228218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.230410 (+ 2192us) server_negotiation.cc:407] Connection header received
0619 02:53:55.230504 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.230505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.230517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.230545 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.230824 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48650: BlockingRecv error: recv got EOF from 127.0.0.1:48650 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 02:53:55.233373 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.232569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.21.164.253:46733)
0619 02:53:55.232603 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.232604 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.232829 (+ 225us) server_negotiation.cc:407] Connection header received
0619 02:53:55.232935 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.232935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.232955 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.232983 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.233324 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58922: BlockingRecv error: recv got EOF from 127.0.0.1:58922 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 02:53:55.236554 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.235742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.21.164.253:46733)
0619 02:53:55.235774 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.235776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.236064 (+ 288us) server_negotiation.cc:407] Connection header received
0619 02:53:55.236152 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.236153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.236163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.236200 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.236501 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58926: BlockingRecv error: recv got EOF from 127.0.0.1:58926 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 02:53:55.240557 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.239830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.21.164.253:46733)
0619 02:53:55.239864 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.239866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.240067 (+ 201us) server_negotiation.cc:407] Connection header received
0619 02:53:55.240157 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.240158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.240168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.240208 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.240510 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58930: BlockingRecv error: recv got EOF from 127.0.0.1:58930 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 02:53:55.246595 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.245816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.21.164.253:46733)
0619 02:53:55.245854 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.245856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.246074 (+ 218us) server_negotiation.cc:407] Connection header received
0619 02:53:55.246169 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.246170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.246180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.246220 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.246543 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58946: BlockingRecv error: recv got EOF from 127.0.0.1:58946 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 02:53:55.256826 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.255936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.21.164.253:46733)
0619 02:53:55.255976 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.255978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.256233 (+ 255us) server_negotiation.cc:407] Connection header received
0619 02:53:55.256336 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.256337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.256349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.256378 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.256767 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58950: BlockingRecv error: recv got EOF from 127.0.0.1:58950 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 02:53:55.275242 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.274343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.21.164.253:46733)
0619 02:53:55.274385 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.274387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.274648 (+ 261us) server_negotiation.cc:407] Connection header received
0619 02:53:55.274750 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.274751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.274764 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.274792 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.275173 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58960: BlockingRecv error: recv got EOF from 127.0.0.1:58960 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 02:53:55.309710 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.308779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.21.164.253:46733)
0619 02:53:55.308822 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.308824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.309090 (+ 266us) server_negotiation.cc:407] Connection header received
0619 02:53:55.309208 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.309209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.309220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.309250 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.309633 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58974: BlockingRecv error: recv got EOF from 127.0.0.1:58974 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 02:53:55.375873 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.375210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.21.164.253:46733)
0619 02:53:55.375298 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.375300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.375329 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:53:55.375420 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.375421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.375432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.375483 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.375808 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58990: BlockingRecv error: recv got EOF from 127.0.0.1:58990 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 02:53:55.505590 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.504946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.21.164.253:46733)
0619 02:53:55.505017 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.505019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.505093 (+ 74us) server_negotiation.cc:407] Connection header received
0619 02:53:55.505168 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.505169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.505181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.505240 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.505540 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59006: BlockingRecv error: recv got EOF from 127.0.0.1:59006 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 02:53:55.763437 12032 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:55.762798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.21.164.253:46733)
0619 02:53:55.762885 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:55.762887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:55.762900 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:53:55.762943 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:55.762944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:55.762955 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:55.762998 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:55.763388 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59018: BlockingRecv error: recv got EOF from 127.0.0.1:59018 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:53:56.136202 12122 catalog_manager.cc:1259] Loaded cluster ID: a8412454340a4183859f08fa204b3ef8
I20260619 02:53:56.136286 12122 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83: loading cluster ID for follower catalog manager: success
I20260619 02:53:56.137205 12122 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83: acquiring CA information for follower catalog manager: success
I20260619 02:53:56.137497 12122 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2d8b15a51124d01bcd57348421b8d83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:53:56.199542 12168 catalog_manager.cc:1259] Loaded cluster ID: a8412454340a4183859f08fa204b3ef8
I20260619 02:53:56.199618 12168 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc: loading cluster ID for follower catalog manager: success
I20260619 02:53:56.200441 12168 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc: acquiring CA information for follower catalog manager: success
I20260619 02:53:56.200735 12168 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af5502c0bb1c4faf9ee421dbb4686bdc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:53:56.279410 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:56.276590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.21.164.253:46733)
0619 02:53:56.276724 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:56.276726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:56.276736 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:53:56.276764 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:56.276765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:56.276821 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:56.276885 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:56.277025 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:56.277541 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:56.278385 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:56.278584 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:56.278638 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:56.278790 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:56.278913 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:56.278996 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:56.279299 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 02:53:56.279340 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260619 02:53:56.283551 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:56.281090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.21.164.252:36185)
0619 02:53:56.281199 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:56.281200 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:56.281272 (+ 72us) server_negotiation.cc:407] Connection header received
0619 02:53:56.281313 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:56.281314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:56.281348 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:56.281421 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:56.281503 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:56.282001 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:56.282780 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:56.282975 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:56.283027 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:56.283166 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:56.283279 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:56.283333 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:56.283441 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 02:53:56.283462 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260619 02:53:56.288009 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:56.285155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.21.164.253:46733)
0619 02:53:56.285192 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:56.285194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:56.285423 (+ 229us) server_negotiation.cc:407] Connection header received
0619 02:53:56.285510 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:56.285511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:56.285523 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:56.285570 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:56.285703 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:56.286465 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:56.287273 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:56.287559 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:56.287615 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:56.287736 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:56.287767 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:56.287812 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:56.287912 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 02:53:56.287935 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 02:53:56.288161 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:56.285348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.21.164.252:36185)
0619 02:53:56.285402 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:56.285404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:56.285538 (+ 134us) server_negotiation.cc:407] Connection header received
0619 02:53:56.285586 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:56.285587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:56.285595 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:56.285641 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:56.285772 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:56.286470 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:56.287473 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:56.287741 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:56.287798 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:56.287932 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:56.287960 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:56.287987 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:56.288066 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 02:53:56.288085 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 02:53:56.288354 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:56.285129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.21.164.254:46165)
0619 02:53:56.285262 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:56.285264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:56.285401 (+ 137us) server_negotiation.cc:407] Connection header received
0619 02:53:56.285481 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:56.285483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:56.285534 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:56.285606 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:56.285790 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:56.286259 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:56.287226 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:56.287476 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:56.287540 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:56.287713 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:56.287816 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:56.287875 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:56.288251 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 02:53:56.288277 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260619 02:53:56.295236 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 11885
I20260619 02:53:56.299659 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 11960
I20260619 02:53:56.303742 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 12035
2026-06-19T02:53:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2721 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:53:56 dist-test-slave-jt0t krb5kdc[12253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:53:56 dist-test-slave-jt0t krb5kdc[12253](info): set up 2 sockets
Jun 19 02:53:56 dist-test-slave-jt0t krb5kdc[12253](info): commencing operation
krb5kdc: starting...
W20260619 02:53:57.260546 22163 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/test-user.keytab.
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:53:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.164.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
Entry for principal kudu/127.21.164.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab.
I20260619 02:53:57.373343 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:37299
--webserver_interface=127.21.164.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:37299,127.21.164.253:37451,127.21.164.252:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:53:57.453008 12266 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:53:57.453226 12266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:57.453243 12266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:57.453261 12266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:57.454416 12266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:57.454448 12266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:57.454463 12266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:57.454474 12266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:57.454490 12266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:57.455986 12266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:37299,127.21.164.253:37451,127.21.164.252:43069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.254.keytab
--principal=kudu/127.21.164.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.254:37299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.164.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:57.456197 12266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:57.456382 12266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:53:57.461130 12266 server_base.cc:1034] running on GCE node
W20260619 02:53:57.461089 12271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:57.461112 12272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:57.461076 12274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:57.461537 12266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:57.461819 12266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:57.462965 12266 hybrid_clock.cc:648] HybridClock initialized: now 1781837637462954 us; error 29 us; skew 500 ppm
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:57.465301 12266 init.cc:377] Logged in from keytab as kudu/127.21.164.254@KRBTEST.COM (short username kudu)
I20260619 02:53:57.465673 12266 webserver.cc:460] Webserver started at http://127.21.164.254:36853/ using document root <none> and password file <none>
I20260619 02:53:57.465852 12266 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:57.465891 12266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:57.465981 12266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:57.466969 12266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/data/instance:
uuid: "2362c2ea11cd474080dc7fa2e462cbfb"
format_stamp: "Formatted at 2026-06-19 02:53:57 on dist-test-slave-jt0t"
server_key: "d84582ca352e41bf5f7f6a20f601e52a"
server_key_iv: "35d1c6ebfecb2e3fa2ccaba683df88c4"
server_key_version: "encryptionkey@0"
I20260619 02:53:57.467255 12266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/wal/instance:
uuid: "2362c2ea11cd474080dc7fa2e462cbfb"
format_stamp: "Formatted at 2026-06-19 02:53:57 on dist-test-slave-jt0t"
server_key: "d84582ca352e41bf5f7f6a20f601e52a"
server_key_iv: "35d1c6ebfecb2e3fa2ccaba683df88c4"
server_key_version: "encryptionkey@0"
I20260619 02:53:57.468461 12266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 02:53:57.469167 12281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:57.469292 12266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:57.469367 12266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/wal
uuid: "2362c2ea11cd474080dc7fa2e462cbfb"
format_stamp: "Formatted at 2026-06-19 02:53:57 on dist-test-slave-jt0t"
server_key: "d84582ca352e41bf5f7f6a20f601e52a"
server_key_iv: "35d1c6ebfecb2e3fa2ccaba683df88c4"
server_key_version: "encryptionkey@0"
I20260619 02:53:57.469432 12266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:57.493072 12266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:57.495215 12266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:57.495344 12266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:57.499063 12266 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.254:37299
I20260619 02:53:57.499107 12333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.254:37299 every 8 connection(s)
I20260619 02:53:57.499428 12266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/data/info.pb
I20260619 02:53:57.500221 12334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:57.501780 12334 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:57.508162 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 12266
I20260619 02:53:57.508291 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-0/wal/instance
I20260619 02:53:57.508497 22163 external_mini_cluster.cc:1428] Setting key f26fa8e01f046b957555400adc2bcf00
I20260619 02:53:57.511446 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.502465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.21.164.254:37299)
0619 02:53:57.502627 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.502632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.502701 (+ 69us) server_negotiation.cc:407] Connection header received
0619 02:53:57.503222 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.503223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.503243 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.503328 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.503829 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.504622 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.505532 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.505843 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.505918 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.507993 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.508002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.508002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.508010 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.509774 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.510273 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.510276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.510277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.510368 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.510594 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.510595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.510595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.510811 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.510873 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.511115 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.511286 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260619 02:53:57.511451 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.502432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:37299 (local address 127.0.0.1:37802)
0619 02:53:57.502583 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.502617 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:57.503095 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:57.503339 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:57.503341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:57.503579 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:57.503760 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.503764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.504775 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.504776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.505460 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.505462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.505944 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.505945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.506011 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.506508 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:57.506518 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:57.507878 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:57.509875 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.509882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.509884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.510154 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.510431 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.510432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.510433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.510540 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.510869 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:57.510874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:57.510950 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 02:53:57.511049 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:53:57.511221 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260619 02:53:57.512179 12334 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:53:57.512526 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.512399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:37451 (local address 127.0.0.1:59366)
0619 02:53:57.512446 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.512499 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 02:53:57.512714 12284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37451: connect: Connection refused (error 111)
I20260619 02:53:57.513109 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.513022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:37451 (local address 127.0.0.1:59380)
0619 02:53:57.513059 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.513087 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 02:53:57.513206 12334 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:37451: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37451: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.21.164.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab.
Entry for principal kudu/127.21.164.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab.
I20260619 02:53:57.532078 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.253:37451
--webserver_interface=127.21.164.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab
--principal=kudu/127.21.164.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:37299,127.21.164.253:37451,127.21.164.252:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:53:57.562426 12334 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } attempt: 1
I20260619 02:53:57.562966 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.562821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:37451 (local address 127.0.0.1:59384)
0619 02:53:57.562883 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.562930 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 02:53:57.563355 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.563268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:37451 (local address 127.0.0.1:59390)
0619 02:53:57.563309 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.563334 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:53:57.563434 12334 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:37451: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37451: connect: Connection refused (error 111)
W20260619 02:53:57.612629 12341 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:53:57.612869 12341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:57.612883 12341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:57.612905 12341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:53:57.614092 12341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:57.614131 12341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:57.614145 12341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:57.614156 12341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:57.614169 12341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:57.615641 12341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:37299,127.21.164.253:37451,127.21.164.252:43069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.253.keytab
--principal=kudu/127.21.164.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.253:37451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.164.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:57.615864 12341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:57.616071 12341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:57.620499 12349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:57.620507 12347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:57.620517 12346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:57.620675 12341 server_base.cc:1034] running on GCE node
I20260619 02:53:57.620971 12341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:57.621279 12341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:57.622426 12341 hybrid_clock.cc:648] HybridClock initialized: now 1781837637622413 us; error 30 us; skew 500 ppm
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:57.624855 12341 init.cc:377] Logged in from keytab as kudu/127.21.164.253@KRBTEST.COM (short username kudu)
I20260619 02:53:57.625267 12341 webserver.cc:460] Webserver started at http://127.21.164.253:35467/ using document root <none> and password file <none>
I20260619 02:53:57.625485 12341 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:57.625533 12341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:57.625660 12341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:57.626757 12341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/data/instance:
uuid: "934ec89843ef41b981ae864b0e68142e"
format_stamp: "Formatted at 2026-06-19 02:53:57 on dist-test-slave-jt0t"
server_key: "bb14162cd6568bbe43015799ef0d172e"
server_key_iv: "eaad6a53669d09a5d949726789724f3a"
server_key_version: "encryptionkey@0"
I20260619 02:53:57.627085 12341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/wal/instance:
uuid: "934ec89843ef41b981ae864b0e68142e"
format_stamp: "Formatted at 2026-06-19 02:53:57 on dist-test-slave-jt0t"
server_key: "bb14162cd6568bbe43015799ef0d172e"
server_key_iv: "eaad6a53669d09a5d949726789724f3a"
server_key_version: "encryptionkey@0"
I20260619 02:53:57.628311 12341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 02:53:57.629073 12356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:57.629217 12341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:57.629297 12341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/wal
uuid: "934ec89843ef41b981ae864b0e68142e"
format_stamp: "Formatted at 2026-06-19 02:53:57 on dist-test-slave-jt0t"
server_key: "bb14162cd6568bbe43015799ef0d172e"
server_key_iv: "eaad6a53669d09a5d949726789724f3a"
server_key_version: "encryptionkey@0"
I20260619 02:53:57.629354 12341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:57.631572 12334 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } attempt: 2
I20260619 02:53:57.631968 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.631853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:37451 (local address 127.0.0.1:59400)
0619 02:53:57.631904 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.631940 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 02:53:57.632287 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.632200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:37451 (local address 127.0.0.1:59406)
0619 02:53:57.632239 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.632263 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 02:53:57.632367 12334 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.253:37451: Network error: Client connection negotiation failed: client connection to 127.21.164.253:37451: connect: Connection refused (error 111)
I20260619 02:53:57.640532 12341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:57.642457 12341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:57.642602 12341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:57.646260 12341 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.253:37451
I20260619 02:53:57.646303 12408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.253:37451 every 8 connection(s)
I20260619 02:53:57.646616 12341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/data/info.pb
I20260619 02:53:57.647436 12409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:57.649001 12409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:57.656662 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 12341
I20260619 02:53:57.656776 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-1/wal/instance
I20260619 02:53:57.656961 22163 external_mini_cluster.cc:1428] Setting key 913e3c06fc7ca194692b7db3c5273d04
I20260619 02:53:57.657344 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.649712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.21.164.254:37299)
0619 02:53:57.649770 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.649772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.649951 (+ 179us) server_negotiation.cc:407] Connection header received
0619 02:53:57.650393 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.650394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.650404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.650466 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.651046 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.651533 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.652440 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.652700 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.652749 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.654719 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.654721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.654721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.654730 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.656001 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.656386 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.656387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.656387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.656440 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.656677 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.656678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.656678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.656821 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.656886 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.657169 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.657253 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 02:53:57.657516 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.649640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:37299 (local address 127.0.0.1:37810)
0619 02:53:57.649816 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.649857 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:57.650314 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:57.650536 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:57.650540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:57.650779 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:57.650964 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.650968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.651596 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.651597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.652371 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.652373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.652767 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.652768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.652867 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.653280 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:57.653287 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:57.654621 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:57.656080 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.656088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.656090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.656320 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.656494 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.656495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.656495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.656604 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.656875 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:57.656882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:57.657035 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 02:53:57.657147 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:53:57.657304 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260619 02:53:57.658026 12409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.164.252@KRBTEST.COM; defaulting to no policy
I20260619 02:53:57.664983 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.658281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:37451 (local address 127.0.0.1:59414)
0619 02:53:57.658338 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.658349 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:57.658393 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:57.658623 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:57.658623 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:57.658742 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:57.658793 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.658794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.659584 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.659585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.660267 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.660269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.660808 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.660809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.660882 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.661246 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:57.661250 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:57.662429 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:57.664008 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.664008 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.664009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.664161 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.664337 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.664338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.664338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.664390 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.664637 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:57.664638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:57.664699 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 02:53:57.664768 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 02:53:57.664871 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 02:53:57.665041 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.658359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.21.164.253:37451)
0619 02:53:57.658505 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.658509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.658526 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:53:57.658544 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.658546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.658558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.658614 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.658854 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.659514 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.660355 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.660737 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.660799 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.662501 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.662509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.662510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.662517 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.663935 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.664233 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.664237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.664237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.664284 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.664437 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.664438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.664439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.664578 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.664674 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.664801 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.664923 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":121,"thread_start_us":98,"threads_started":1}
I20260619 02:53:57.665721 12409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:53:57.666091 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.665947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:53988)
0619 02:53:57.666017 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.666065 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:53:57.666250 12359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
I20260619 02:53:57.666671 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.666577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:53992)
0619 02:53:57.666617 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.666646 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:53:57.666867 12409 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:43069: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.164.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.164.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab.
Entry for principal kudu/127.21.164.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab.
I20260619 02:53:57.681649 22163 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1oh3JA/build/release/bin/kudu
/tmp/dist-test-task1oh3JA/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.252:43069
--webserver_interface=127.21.164.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab
--principal=kudu/127.21.164.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.164.212:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.164.254:37299,127.21.164.253:37451,127.21.164.252:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:53:57.716095 12409 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } attempt: 1
I20260619 02:53:57.716614 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.716436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:53994)
0619 02:53:57.716510 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.716568 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 02:53:57.717060 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.716957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:54006)
0619 02:53:57.716998 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.717030 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 02:53:57.717157 12409 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:43069: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
I20260619 02:53:57.723500 12334 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } attempt: 3
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
I20260619 02:53:57.730232 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.723854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.21.164.253:37451)
0619 02:53:57.723913 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.723915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.723925 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:53:57.723952 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.723953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.723966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.724009 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.724329 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.724862 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.725565 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.725806 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.725872 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.727856 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.727858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.727858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.727865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.729190 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.729563 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.729563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.729564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.729631 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.729805 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.729806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.729807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.729910 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.729955 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.730096 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.730156 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 02:53:57.730264 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.723793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:37451 (local address 127.0.0.1:59428)
0619 02:53:57.723844 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.723855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:57.723900 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:57.724020 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:57.724020 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:57.724174 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:57.724256 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.724259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.724929 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.724930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.725491 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.725494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.725873 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.725875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.725944 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.726432 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:57.726437 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:57.727754 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:57.729269 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.729270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.729271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.729488 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.729689 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.729690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.729691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.729753 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.729961 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:57.729961 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:57.730012 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 02:53:57.730077 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:53:57.730169 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 02:53:57.730743 12334 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:53:57.731066 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.730960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:54012)
0619 02:53:57.731006 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.731041 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 02:53:57.731355 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.731277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:54026)
0619 02:53:57.731312 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.731336 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 02:53:57.731439 12334 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:43069: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
I20260619 02:53:57.762584 12334 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } attempt: 1
W20260619 02:53:57.762701 12416 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 02:53:57.762976 12416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:53:57.763012 12416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:53:57.763051 12416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 02:53:57.763083 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.762951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:54036)
0619 02:53:57.763016 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.763049 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:53:57.763417 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.763329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:54050)
0619 02:53:57.763368 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.763394 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 02:53:57.763511 12334 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:43069: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
W20260619 02:53:57.764536 12416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:53:57.764586 12416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:53:57.764605 12416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:53:57.764631 12416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:53:57.764664 12416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:53:57.766194 12416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.164.212:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.164.254:37299,127.21.164.253:37451,127.21.164.252:43069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/krb5kdc/kudu_127.21.164.252.keytab
--principal=kudu/127.21.164.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.164.252:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.164.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 01:55:14 UTC on ccycloud.quasar-liqyol.root.comops.site
build id 4454980
I20260619 02:53:57.766451 12416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:53:57.766660 12416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:53:57.771298 12424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:53:57.771322 12421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:57.771489 12416 server_base.cc:1034] running on GCE node
W20260619 02:53:57.771322 12422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:53:57.771795 12416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:53:57.772065 12416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:53:57.773201 12416 hybrid_clock.cc:648] HybridClock initialized: now 1781837637773187 us; error 30 us; skew 500 ppm
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:53:57.775393 12416 init.cc:377] Logged in from keytab as kudu/127.21.164.252@KRBTEST.COM (short username kudu)
I20260619 02:53:57.775831 12416 webserver.cc:460] Webserver started at http://127.21.164.252:44737/ using document root <none> and password file <none>
I20260619 02:53:57.776034 12416 fs_manager.cc:359] Metadata directory not provided
I20260619 02:53:57.776080 12416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:53:57.776203 12416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:53:57.777264 12416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/data/instance:
uuid: "f7d50a09f7414f91aeed7029e2f4e14e"
format_stamp: "Formatted at 2026-06-19 02:53:57 on dist-test-slave-jt0t"
server_key: "c5a5f54a2f204afef5abf51781432f0e"
server_key_iv: "8ca2c0211fcf416d48b8dde48dec570e"
server_key_version: "encryptionkey@0"
I20260619 02:53:57.777603 12416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/wal/instance:
uuid: "f7d50a09f7414f91aeed7029e2f4e14e"
format_stamp: "Formatted at 2026-06-19 02:53:57 on dist-test-slave-jt0t"
server_key: "c5a5f54a2f204afef5abf51781432f0e"
server_key_iv: "8ca2c0211fcf416d48b8dde48dec570e"
server_key_version: "encryptionkey@0"
I20260619 02:53:57.778861 12416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 02:53:57.779542 12431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:53:57.779681 12416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 02:53:57.779758 12416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/data,/tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/wal
uuid: "f7d50a09f7414f91aeed7029e2f4e14e"
format_stamp: "Formatted at 2026-06-19 02:53:57 on dist-test-slave-jt0t"
server_key: "c5a5f54a2f204afef5abf51781432f0e"
server_key_iv: "8ca2c0211fcf416d48b8dde48dec570e"
server_key_version: "encryptionkey@0"
I20260619 02:53:57.779829 12416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:53:57.785316 12409 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } attempt: 2
I20260619 02:53:57.785733 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.785595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:54060)
0619 02:53:57.785654 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.785698 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 02:53:57.786070 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.785983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:54072)
0619 02:53:57.786022 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.786049 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 02:53:57.786149 12409 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.164.252:43069: Network error: Client connection negotiation failed: client connection to 127.21.164.252:43069: connect: Connection refused (error 111)
I20260619 02:53:57.802615 12416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:53:57.804854 12416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:53:57.804998 12416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:53:57.808961 12416 rpc_server.cc:307] RPC server started. Bound to: 127.21.164.252:43069
I20260619 02:53:57.808995 12483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.164.252:43069 every 8 connection(s)
I20260619 02:53:57.809319 12416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/data/info.pb
I20260619 02:53:57.810132 12484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:53:57.811652 12484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:53:57.816417 22163 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1oh3JA/build/release/bin/kudu as pid 12416
I20260619 02:53:57.816520 22163 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1oh3JA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837375929108-22163-0/minicluster-data/master-2/wal/instance
I20260619 02:53:57.816772 22163 external_mini_cluster.cc:1428] Setting key ef8fdf60050a60d4df81df3dab690524
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
I20260619 02:53:57.820446 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.812337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.21.164.254:37299)
0619 02:53:57.812401 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.812404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.812589 (+ 185us) server_negotiation.cc:407] Connection header received
0619 02:53:57.813124 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.813124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.813135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.813187 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.813768 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.814228 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.815216 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.815415 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.815482 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.817855 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.817857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.817857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.817865 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.819154 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.819525 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.819526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.819526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.819581 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.819803 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.819804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.819804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.819915 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.819969 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.820256 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.820350 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 02:53:57.820576 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.812311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.254:37299 (local address 127.0.0.1:37822)
0619 02:53:57.812475 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.812513 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:57.813031 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:57.813251 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:57.813255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:57.813502 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:57.813698 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.813702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.814312 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.814313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.815107 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.815108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.815498 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.815499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.815605 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.816081 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:57.816088 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:57.817727 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:57.819231 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.819238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.819240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.819457 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.819632 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.819633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.819633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.819740 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.819968 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:57.819974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:57.820110 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 02:53:57.820219 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 02:53:57.820374 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260619 02:53:57.821161 12484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
I20260619 02:53:57.825234 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.817184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.21.164.252:43069)
0619 02:53:57.817357 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.817427 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.817450 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:53:57.817470 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.817472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.817489 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.817551 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.818188 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.818870 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.819566 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.819870 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.819927 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.822367 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.822374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.822374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.822381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.823950 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.824312 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.824315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.824315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.824375 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.824562 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.824563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.824563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.824764 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.824947 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.825034 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.825109 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260619 02:53:57.828161 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.821498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.21.164.253:37451)
0619 02:53:57.821540 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.821542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.821562 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:53:57.821585 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.821585 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.821593 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.821627 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.821931 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.822471 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.823325 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.823538 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.823594 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.825430 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.825431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.825432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.825439 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.826977 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.827339 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.827340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.827340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.827406 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.827611 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.827612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.827612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.827731 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.827870 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.828001 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.828072 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 02:53:57.828161 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.821441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.253:37451 (local address 127.0.0.1:59436)
0619 02:53:57.821475 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.821486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:57.821525 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:57.821633 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:57.821634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:57.821772 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:57.821833 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.821835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.822540 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.822541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.823225 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.823227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.823606 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.823606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.823675 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.824000 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:57.824005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:57.825353 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:57.827049 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.827049 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.827050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.827251 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.827496 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.827497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.827497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.827558 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.827855 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:57.827856 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:57.827902 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:53:57.827969 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 02:53:57.828060 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 02:53:57.828572 12484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.254@KRBTEST.COM
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.252@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
I20260619 02:53:57.833714 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.826681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.21.164.254:37299)
0619 02:53:57.826727 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.826729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.826736 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:53:57.826756 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.826757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.826765 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.826812 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.827368 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.827822 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.828706 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.828947 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.829011 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.831049 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.831051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.831051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.831059 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.832498 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.832889 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.832890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.832891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.832950 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.833154 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.833155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.833155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.833297 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.833503 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.833542 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.833615 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 02:53:57.834867 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.828852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.21.164.252:43069)
0619 02:53:57.828882 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.828883 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.828904 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:53:57.828958 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.828958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.828966 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.829011 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.829227 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.829684 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.830334 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.830525 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.830574 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.832284 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.832285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.832286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.832292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.833782 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.834092 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.834093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.834093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.834139 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.834328 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.834328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.834329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.834454 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.834505 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.834641 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.834773 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":14}
I20260619 02:53:57.834874 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.828821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:54092)
0619 02:53:57.828861 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.828871 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:57.828916 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:57.829015 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:57.829016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:57.829138 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:57.829166 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.829167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.829745 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.829745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.830254 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.830255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.830586 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.830587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.830637 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.830949 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:57.830953 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:57.832176 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:57.833856 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.833857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.833857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.834031 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.834190 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.834191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.834191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.834264 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.834504 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:57.834505 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:57.834552 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 02:53:57.834623 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 02:53:57.834735 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":2}
I20260619 02:53:57.836334 12484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e: Bootstrap starting.
I20260619 02:53:57.837368 12484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:57.837622 12484 log.cc:824] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.164.253@KRBTEST.COM
I20260619 02:53:57.838656 12334 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } attempt: 2
I20260619 02:53:57.838848 12484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e: No bootstrap required, opened a new log
I20260619 02:53:57.840883 12484 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } }
I20260619 02:53:57.841040 12484 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d50a09f7414f91aeed7029e2f4e14e, State: Initialized, Role: FOLLOWER
I20260619 02:53:57.841135 12484 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } }
I20260619 02:53:57.841472 12484 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:57.841418 12490 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } } }
I20260619 02:53:57.841852 12490 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:57.841943 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.834372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.21.164.253:37451)
0619 02:53:57.834436 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.834438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.834445 (+ 7us) server_negotiation.cc:407] Connection header received
0619 02:53:57.834467 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.834468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.834476 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.834519 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.835135 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.835769 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.836396 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.836707 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.836776 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.839255 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.839257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.839257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.839265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.840622 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.840996 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.840997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.840998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.841059 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.841281 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.841283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.841283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.841409 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.841565 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.841779 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.841853 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.254@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
I20260619 02:53:57.843480 22163 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 02:53:57.843204 12501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:57.844838 12501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:57.844594 12308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2362c2ea11cd474080dc7fa2e462cbfb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37826
I20260619 02:53:57.845654 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.838913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:54100)
0619 02:53:57.838964 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.838975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:57.839011 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:57.839141 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:57.839142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:57.839295 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:57.839334 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.839335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.840049 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.840050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.840852 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.840856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.841319 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.841320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.841379 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.841721 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:57.841726 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:57.843166 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:57.844604 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.844605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.844605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.844840 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.845021 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.845022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.845022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.845081 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.845369 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:57.845370 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:57.845418 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 02:53:57.845476 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 02:53:57.845563 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 02:53:57.846132 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.838956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.21.164.252:43069)
0619 02:53:57.838991 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.838993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.839001 (+ 8us) server_negotiation.cc:407] Connection header received
0619 02:53:57.839055 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.839055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.839065 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.839122 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.839399 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.839982 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.840935 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.841212 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.841268 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.843269 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.843270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.843271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.843277 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.844533 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.844918 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.844919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.844919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.844972 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.845142 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.845143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.845143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.845284 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.845343 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.845960 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.846043 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:53:57.847558 12334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb: Bootstrap starting.
I20260619 02:53:57.845181 12416 master_runner.cc:186] Error getting master registration for 127.21.164.254:37299: OK, instance_id { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" instance_seqno: 1781837637498046 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:53:57.847908 12416 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:53:57.848204 12334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:57.848409 12334 log.cc:824] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:57.849112 12334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb: No bootstrap required, opened a new log
I20260619 02:53:57.850064 12334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } }
I20260619 02:53:57.850171 12334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2362c2ea11cd474080dc7fa2e462cbfb, State: Initialized, Role: FOLLOWER
I20260619 02:53:57.850244 12334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } }
I20260619 02:53:57.850471 12504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } } }
I20260619 02:53:57.850541 12308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 02:53:57.850518 12334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:57.850574 12504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:57.850622 12308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:57.851414 12308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } }
I20260619 02:53:57.851639 12308 leader_election.cc:288] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [CANDIDATE]: Term 1 election: Requested vote from peers 934ec89843ef41b981ae864b0e68142e (127.21.164.253:37451), f7d50a09f7414f91aeed7029e2f4e14e (127.21.164.252:43069)
W20260619 02:53:57.852273 12515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 02:53:57.852273 12458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f7d50a09f7414f91aeed7029e2f4e14e"
I20260619 02:53:57.852409 12458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [term 0 FOLLOWER]: Advancing to term 1
W20260619 02:53:57.852380 12515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:57.853657 12458 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2362c2ea11cd474080dc7fa2e462cbfb in term 1.
I20260619 02:53:57.853879 12284 leader_election.cc:302] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2362c2ea11cd474080dc7fa2e462cbfb, f7d50a09f7414f91aeed7029e2f4e14e; no voters:
I20260619 02:53:57.853996 12504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:53:57.854115 12504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [term 1 LEADER]: Becoming Leader. State: Replica: 2362c2ea11cd474080dc7fa2e462cbfb, State: Running, Role: LEADER
I20260619 02:53:57.854205 12504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } }
I20260619 02:53:57.854588 12504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2362c2ea11cd474080dc7fa2e462cbfb. Latest consensus state: current_term: 1 leader_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } } }
I20260619 02:53:57.854676 12504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:57.854869 12523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:53:57.855058 12523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:53:57.852990 12266 master_runner.cc:186] Error getting master registration for 127.21.164.253:37451: OK, instance_id { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" instance_seqno: 1781837637645223 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:53:57.855849 12266 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:53:57.857617 12458 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [term 1 FOLLOWER]: Refusing update from remote peer 2362c2ea11cd474080dc7fa2e462cbfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:57.857975 12504 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:53:57.859241 12490 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2362c2ea11cd474080dc7fa2e462cbfb. Latest consensus state: current_term: 1 leader_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } } }
I20260619 02:53:57.859330 12490 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:57.866700 12504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } } }
I20260619 02:53:57.866809 12504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [sys.catalog]: This master's current role is: LEADER
I20260619 02:53:57.866879 12524 mvcc.cc:205] Tried to move back new op lower bound from 7298406964660678656 to 7298406964651511808. Current Snapshot: MvccSnapshot[applied={T|T < 7298406964660678656}]
I20260619 02:53:57.867321 12490 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } } }
I20260619 02:53:57.867411 12490 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:57.867585 12528 mvcc.cc:205] Tried to move back new op lower bound from 7298406964660678656 to 7298406964651511808. Current Snapshot: MvccSnapshot[applied={T|T < 7298406964660678656}]
I20260619 02:53:57.872020 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.857443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.21.164.252:43069)
0619 02:53:57.857480 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.857481 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.864907 (+ 7426us) server_negotiation.cc:407] Connection header received
0619 02:53:57.864958 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.864959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.864969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.865013 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.865783 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.866448 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.867371 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.867632 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.867672 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.869172 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.869174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.869174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.869180 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.870759 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.871145 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.871146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.871147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.871198 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.871385 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.871386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.871386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.871506 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.871575 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.871822 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.871926 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 02:53:57.873234 12523 catalog_manager.cc:1347] Generated new cluster ID: 55178849bd8e4f609a7fa3dd7d08a83d
I20260619 02:53:57.873287 12523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:53:57.877346 12409 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } attempt: 3
I20260619 02:53:57.881932 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.856951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.21.164.254:37299)
0619 02:53:57.856989 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.856991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.864812 (+ 7821us) server_negotiation.cc:407] Connection header received
0619 02:53:57.864928 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.864930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.864942 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.865003 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.868956 (+ 3953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.869701 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.873945 (+ 4244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.874256 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.874296 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.878671 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.878672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.878672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.878680 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.880135 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.880818 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.880819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.880819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.880878 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.881178 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.881179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.881180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.881325 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.881405 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.881739 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.881833 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Jun 19 02:53:57 dist-test-slave-jt0t krb5kdc[12253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837637, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.164.253@KRBTEST.COM for kudu/127.21.164.252@KRBTEST.COM
I20260619 02:53:57.884444 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.857205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.21.164.253:37451)
0619 02:53:57.857249 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.857251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.864812 (+ 7561us) server_negotiation.cc:407] Connection header received
0619 02:53:57.865033 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.865034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.865044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.865071 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.865996 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.866729 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.871337 (+ 4608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.871661 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.871700 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.877083 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.877085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.877085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.877094 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.880727 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.883427 (+ 2700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.883428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.883429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.883478 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.883931 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.883931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.883932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.884047 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.884114 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.884282 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.884352 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:53:57.887141 12523 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:53:57.887226 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.877618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.164.252:43069 (local address 127.0.0.1:54112)
0619 02:53:57.877723 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 02:53:57.877733 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 02:53:57.877778 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:53:57.877932 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:53:57.877933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:53:57.878113 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:53:57.878154 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.878156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.879160 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.879161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.880020 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:53:57.880023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.881190 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.881191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:53:57.881246 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.881573 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:53:57.881578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:53:57.882871 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:53:57.885886 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.885887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.885888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.886106 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.886391 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:53:57.886392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:53:57.886392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:53:57.886461 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:53:57.886918 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:53:57.886919 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:53:57.886978 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 02:53:57.887032 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:53:57.887128 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 02:53:57.887872 12523 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:53:57.888062 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:57.877664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.21.164.252:43069)
0619 02:53:57.877699 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.877701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.877764 (+ 63us) server_negotiation.cc:407] Connection header received
0619 02:53:57.877832 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.877832 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.877841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.877888 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.878386 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.879082 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.880806 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:57.881096 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:57.881145 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:57.882951 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:53:57.882952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:53:57.882952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:53:57.882958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:53:57.885745 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.886191 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.886193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.886193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.886311 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:53:57.886537 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:53:57.886538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:53:57.886538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:53:57.886790 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:53:57.887626 (+ 836us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:57.887697 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 02:53:57.887837 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 02:53:57.889870 12409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e: Bootstrap starting.
I20260619 02:53:57.890601 12409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e: Neither blocks nor log segments found. Creating new log.
I20260619 02:53:57.890890 12409 log.cc:824] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:53:57.892824 12409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e: No bootstrap required, opened a new log
I20260619 02:53:57.894846 12409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } }
I20260619 02:53:57.894987 12409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934ec89843ef41b981ae864b0e68142e, State: Initialized, Role: FOLLOWER
I20260619 02:53:57.895061 12409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } }
I20260619 02:53:57.895380 12409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:53:57.896296 12523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb: Generated new TSK 0
I20260619 02:53:57.896462 12523 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:53:57.897001 12362 tablet.cc:2369] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:53:57.896597 12383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "934ec89843ef41b981ae864b0e68142e"
I20260619 02:53:57.898892 12382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:53:57.899780 12382 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [term 1 FOLLOWER]: Refusing update from remote peer 2362c2ea11cd474080dc7fa2e462cbfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:53:57.900727 12534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } } }
I20260619 02:53:57.900810 12534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:57.902581 12383 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2362c2ea11cd474080dc7fa2e462cbfb in term 1.
I20260619 02:53:57.902868 12531 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:53:57.903736 12545 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:53:57.903788 12545 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:53:57.904376 12534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2362c2ea11cd474080dc7fa2e462cbfb. Latest consensus state: current_term: 1 leader_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } } }
I20260619 02:53:57.904452 12534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:57.904485 12534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } } }
I20260619 02:53:57.904521 12534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:53:57.912338 12548 mvcc.cc:205] Tried to move back new op lower bound from 7298406964817891328 to 7298406964651511808. Current Snapshot: MvccSnapshot[applied={T|T < 7298406964660678656}]
W20260619 02:53:57.933809 12413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:57.932164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.21.164.253:37451)
0619 02:53:57.932202 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.932203 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.933046 (+ 843us) server_negotiation.cc:407] Connection header received
0619 02:53:57.933158 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.933159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.933171 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.933201 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.933736 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59446: BlockingRecv error: recv got EOF from 127.0.0.1:59446 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 02:53:57.934687 12337 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:57.931888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.21.164.254:37299)
0619 02:53:57.931936 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.931938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.934042 (+ 2104us) server_negotiation.cc:407] Connection header received
0619 02:53:57.934142 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.934144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.934155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.934184 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.934623 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37842: BlockingRecv error: recv got EOF from 127.0.0.1:37842 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 02:53:57.935346 12488 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:57.932613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.21.164.252:43069)
0619 02:53:57.932685 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.932687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.934857 (+ 2170us) server_negotiation.cc:407] Connection header received
0619 02:53:57.934949 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.934950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.934961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.934989 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.935291 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54124: BlockingRecv error: recv got EOF from 127.0.0.1:54124 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 02:53:57.937711 12413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:57.936979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.21.164.253:37451)
0619 02:53:57.937013 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.937014 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.937210 (+ 196us) server_negotiation.cc:407] Connection header received
0619 02:53:57.937324 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.937324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.937343 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.937369 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.937662 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59452: BlockingRecv error: recv got EOF from 127.0.0.1:59452 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 02:53:57.940686 12413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:57.939930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.21.164.253:37451)
0619 02:53:57.939963 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.939964 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.940192 (+ 228us) server_negotiation.cc:407] Connection header received
0619 02:53:57.940292 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.940293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.940303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.940329 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.940616 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59468: BlockingRecv error: recv got EOF from 127.0.0.1:59468 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 02:53:57.945727 12413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:57.944921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.21.164.253:37451)
0619 02:53:57.944956 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.944958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.945197 (+ 239us) server_negotiation.cc:407] Connection header received
0619 02:53:57.945304 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.945305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.945317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.945344 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.945682 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59470: BlockingRecv error: recv got EOF from 127.0.0.1:59470 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 02:53:57.951853 12413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:57.950947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.21.164.253:37451)
0619 02:53:57.950986 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.950989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.951306 (+ 317us) server_negotiation.cc:407] Connection header received
0619 02:53:57.951411 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.951412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.951423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.951452 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.951803 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59482: BlockingRecv error: recv got EOF from 127.0.0.1:59482 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 02:53:57.962126 12413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:57.961213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.21.164.253:37451)
0619 02:53:57.961255 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.961257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.961546 (+ 289us) server_negotiation.cc:407] Connection header received
0619 02:53:57.961650 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.961652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.961663 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.961691 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.962058 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59496: BlockingRecv error: recv got EOF from 127.0.0.1:59496 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 02:53:57.980433 12413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:57.979509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.21.164.253:37451)
0619 02:53:57.979554 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:57.979556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:57.979830 (+ 274us) server_negotiation.cc:407] Connection header received
0619 02:53:57.979937 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:57.979938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:57.979951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:57.979980 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:57.980374 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59502: BlockingRecv error: recv got EOF from 127.0.0.1:59502 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 02:53:58.014374 12413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:58.013709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.21.164.253:37451)
0619 02:53:58.013752 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:58.013754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:58.013882 (+ 128us) server_negotiation.cc:407] Connection header received
0619 02:53:58.013931 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:58.013932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:58.013944 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:58.013994 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:58.014314 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59516: BlockingRecv error: recv got EOF from 127.0.0.1:59516 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 02:53:58.080009 12413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:58.079363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.21.164.253:37451)
0619 02:53:58.079426 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:58.079428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:58.079494 (+ 66us) server_negotiation.cc:407] Connection header received
0619 02:53:58.079543 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:58.079544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:58.079559 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:58.079621 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:58.079953 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59526: BlockingRecv error: recv got EOF from 127.0.0.1:59526 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 02:53:58.209765 12413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:58.209119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.21.164.253:37451)
0619 02:53:58.209203 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:58.209205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:58.209254 (+ 49us) server_negotiation.cc:407] Connection header received
0619 02:53:58.209325 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:58.209325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:58.209336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:58.209381 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:58.209699 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59530: BlockingRecv error: recv got EOF from 127.0.0.1:59530 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:53:58.445297 12607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2362c2ea11cd474080dc7fa2e462cbfb" member_type: VOTER last_known_addr { host: "127.21.164.254" port: 37299 } } peers { permanent_uuid: "934ec89843ef41b981ae864b0e68142e" member_type: VOTER last_known_addr { host: "127.21.164.253" port: 37451 } } peers { permanent_uuid: "f7d50a09f7414f91aeed7029e2f4e14e" member_type: VOTER last_known_addr { host: "127.21.164.252" port: 43069 } } }
I20260619 02:53:58.445431 12607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2362c2ea11cd474080dc7fa2e462cbfb [sys.catalog]: This master's current role is: LEADER
W20260619 02:53:58.467550 12413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:53:58.466973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.21.164.253:37451)
0619 02:53:58.467035 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:58.467037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:58.467046 (+ 9us) server_negotiation.cc:407] Connection header received
0619 02:53:58.467104 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:58.467104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:58.467113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:58.467162 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:53:58.467498 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59538: BlockingRecv error: recv got EOF from 127.0.0.1:59538 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 02:53:58.845342 12501 catalog_manager.cc:1259] Loaded cluster ID: 55178849bd8e4f609a7fa3dd7d08a83d
I20260619 02:53:58.845425 12501 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e: loading cluster ID for follower catalog manager: success
I20260619 02:53:58.846256 12501 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e: acquiring CA information for follower catalog manager: success
I20260619 02:53:58.846560 12501 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7d50a09f7414f91aeed7029e2f4e14e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:53:58.904239 12545 catalog_manager.cc:1259] Loaded cluster ID: 55178849bd8e4f609a7fa3dd7d08a83d
I20260619 02:53:58.904320 12545 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e: loading cluster ID for follower catalog manager: success
I20260619 02:53:58.905325 12545 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e: acquiring CA information for follower catalog manager: success
I20260619 02:53:58.905668 12545 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 934ec89843ef41b981ae864b0e68142e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:53:58.983282 12619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:58.980704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.21.164.253:37451)
0619 02:53:58.980805 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:58.980806 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:58.980867 (+ 61us) server_negotiation.cc:407] Connection header received
0619 02:53:58.980937 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:58.980938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:58.980986 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:58.981049 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:58.981158 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.981652 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.982456 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.982664 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.982764 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:58.982964 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:58.983076 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:58.983136 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:58.983180 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 02:53:58.983201 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":52,"threads_started":1}
I20260619 02:53:58.987466 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:58.985232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.21.164.252:43069)
0619 02:53:58.985356 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:58.985357 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:58.985367 (+ 10us) server_negotiation.cc:407] Connection header received
0619 02:53:58.985402 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:58.985403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:58.985438 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:58.985516 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:58.985647 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.986120 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.986853 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.987052 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.987103 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:58.987208 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:58.987313 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:58.987371 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:58.987384 (+ 13us) server_negotiation.cc:299] Negotiation successful
0619 02:53:58.987401 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260619 02:53:58.991667 12619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:58.989011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.21.164.253:37451)
0619 02:53:58.989053 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:58.989054 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:58.989212 (+ 158us) server_negotiation.cc:407] Connection header received
0619 02:53:58.989274 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:58.989275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:58.989286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:58.989328 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:58.989479 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.990262 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.991048 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.991253 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.991305 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:58.991428 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:58.991459 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:58.991504 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:58.991565 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 02:53:58.991587 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 02:53:58.992096 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:58.989292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.21.164.252:43069)
0619 02:53:58.989335 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:58.989337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:58.989478 (+ 141us) server_negotiation.cc:407] Connection header received
0619 02:53:58.989527 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:58.989528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:58.989538 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:58.989582 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:58.989719 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.990184 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.991343 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.991546 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.991597 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:58.991764 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:58.991795 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:58.991821 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:58.991927 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 02:53:58.991950 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 02:53:58.992403 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:53:58.989008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.21.164.254:37299)
0619 02:53:58.989107 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 02:53:58.989108 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 02:53:58.989170 (+ 62us) server_negotiation.cc:407] Connection header received
0619 02:53:58.989284 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:53:58.989285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:53:58.989339 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:53:58.989424 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:53:58.989531 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.990291 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.991443 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:53:58.991742 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:53:58.991800 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:53:58.992014 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:53:58.992120 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:53:58.992179 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:53:58.992312 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 02:53:58.992343 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":51,"threads_started":1}
I20260619 02:53:58.993978 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 12266
I20260619 02:53:58.999110 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 12341
I20260619 02:53:59.003199 22163 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1oh3JA/build/release/bin/kudu with pid 12416
2026-06-19T02:53:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2699 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5421 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (263080 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST